CN116052171A - Electronic evidence correlation calibration method, device, equipment and storage medium - Google Patents
Electronic evidence correlation calibration method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN116052171A CN116052171A CN202310332705.2A CN202310332705A CN116052171A CN 116052171 A CN116052171 A CN 116052171A CN 202310332705 A CN202310332705 A CN 202310332705A CN 116052171 A CN116052171 A CN 116052171A
- Authority
- CN
- China
- Prior art keywords
- picture
- text
- feature
- features
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000003860 storage Methods 0.000 title claims abstract description 13
- 230000007246 mechanism Effects 0.000 claims abstract description 27
- 239000000470 constituent Substances 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims description 70
- 230000004927 fusion Effects 0.000 claims description 42
- 239000013598 vector Substances 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 16
- 238000013528 artificial neural network Methods 0.000 claims description 12
- 238000013527 convolutional neural network Methods 0.000 claims description 11
- 238000011176 pooling Methods 0.000 claims description 10
- 125000004122 cyclic group Chemical group 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 7
- 230000000306 recurrent effect Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 description 22
- 238000010586 diagram Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000007789 sealing Methods 0.000 description 1
- 238000007711 solidification Methods 0.000 description 1
- 230000008023 solidification Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/18—Extraction of features or characteristics of the image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/19—Recognition using electronic means
- G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
- G06V30/1918—Fusion techniques, i.e. combining data from various sources, e.g. sensor fusion
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The application discloses an electronic evidence correlation calibration method, device, equipment and storage medium, firstly extracting text features of text state electronic evidence and picture features of picture state electronic evidence, then calculating correlation among all constituent elements in the features based on an attention mechanism for each feature in the text features and the picture features, updating the features by utilizing the correlation to obtain updated text features and updated picture features, fusing the updated text features and the updated picture features to obtain fused features, and finally calibrating the correlation of the text state electronic evidence and the picture state electronic evidence according to the fused features. The correlation among the constituent elements of the common-mode features is utilized to update the directly extracted features, so that updated features which can better represent the actual meaning of the electronic evidence are obtained, and more accurate electronic evidence correlation can be marked according to the updated features.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for calibrating correlation of electronic evidence.
Background
With the popularity and popularity of computers and information networks, evidence informatization becomes a trend, electronic evidence becomes an important carrier for transmitting information and recording facts, becomes a decisive evidence for influencing disputes and cases, and the development of storage units with high security and high stability, such as blockchains, improves the reliability and stability of the electronic evidence in aspects of preservation, verification and the like. In addition, the modal richness of the electronic evidence is improved, the public confidence of the electronic evidence can be improved, for example, when the picture type evidence of the target object is stored, the corresponding text type evidence such as contract, form and text is stored, the non-formatted electronic evidence such as picture and video of the target object and the corresponding document file are stored in a combined mode, and the modal richness of the electronic evidence of the target object can be enriched.
However, in the uploading or storing process of the electronic evidence, the situation that the electronic evidence of different modes of the same object is irrelevant may occur, which may cause that the authenticity of the electronic evidence of the object is difficult to judge, and the credibility of the electronic evidence is reduced.
Disclosure of Invention
In view of the above problems, the present application is provided to provide a method, an apparatus, a device, and a storage medium for calibrating correlation of electronic evidence, so as to realize a task of calibrating correlation of multi-mode electronic evidence.
The specific scheme is as follows:
in a first aspect, an electronic evidence correlation calibration method is provided, including:
extracting text features of the text state electronic evidence and extracting picture features of the picture state electronic evidence;
for each feature in the text feature and the picture feature, calculating the correlation among all constituent elements in the feature based on an attention mechanism, and updating the feature by utilizing the correlation to obtain an updated text feature and an updated picture feature;
fusing the updated text features and the updated picture features to obtain fused features;
and calibrating the correlation of the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
In a second aspect, an electronic evidence correlation calibration device is provided, including:
the text feature extraction module is used for extracting text features of the text state electronic evidence;
the picture feature extraction module is used for extracting picture features of the picture state electronic evidence;
the in-mold attention module is used for calculating the correlation among all the constituent elements in the characteristics based on an attention mechanism for each characteristic in the text characteristics and the picture characteristics, and updating the characteristics by utilizing the correlation to obtain updated text characteristics and updated picture characteristics;
the fusion module is used for fusing the updated text characteristics and the updated picture characteristics to obtain fusion characteristics;
and the correlation calibration module is used for calibrating the correlation between the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
In a third aspect, an electronic evidence correlation calibration apparatus is provided, including: a memory and a processor;
the memory is used for storing programs;
the processor is used for executing the program to realize each step of the electronic evidence correlation calibration method.
In a fourth aspect, a storage medium is provided, on which a computer program is stored, which, when being executed by a processor, implements the steps of the above-mentioned method for calibrating correlation of electronic evidence.
By means of the technical scheme, firstly, feature extraction is carried out on electronic evidence of different modes, then the extracted text features and picture features are updated respectively by using an attention mechanism, in the process, correlation among all constituent elements in the mode features is considered, the features are updated by using the correlation, compared with the features directly extracted, the updated features can reflect the true meaning of the electronic evidence, so that more accurate electronic evidence correlation can be calibrated later, the updated text features and the updated picture features are fused, finally, correlation calibration tasks of the electronic evidence of different modes are calibrated according to the fused features, and the accuracy of the calibrated electronic evidence correlation is improved.
In practical application, the correlation calibration can be performed on the electronic evidence to be stored before the multi-mode electronic evidence is stored, the authenticity of the electronic evidence to be stored is determined based on the calibrated correlation, the electronic evidence to be stored is considered to be true when the correlation is larger than a preset true threshold, the electronic evidence to be stored is allowed to be stored, and the electronic evidence to be stored is considered to be false when the correlation is smaller than the preset false threshold, and the storage is reminded or refused, so that the credibility of the stored electronic evidence is improved.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the application. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
fig. 1 is a schematic flow chart of an electronic evidence correlation calibration method provided in an embodiment of the present application;
FIG. 2 is a schematic diagram of a multi-modal heterographic attention network according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a picture feature extraction module according to an embodiment of the present application;
FIG. 4 is a flowchart of another method for calibrating correlation of electronic evidence according to an embodiment of the present application;
FIG. 5 is a schematic diagram of another multi-modal heterographic attention network provided by an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic evidence correlation calibration device according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic evidence correlation calibration device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The application provides an electronic evidence correlation calibration method, device, equipment and storage medium, which can realize the calibration task of correlation of multi-mode electronic evidence.
The scheme can be realized based on the terminal with the data processing capability, and the terminal can be a computer, a server, a cloud end and the like.
Next, referring to fig. 1, the method for calibrating correlation of electronic evidence provided in the present application may include the following steps S101 to S104:
and step S101, extracting text features of the text state electronic evidence and extracting picture features of the picture state electronic evidence.
It should be noted that the photo-state electronic evidence may be a captured picture or video, and for the video evidence, a plurality of frames of images may be extracted, and the extracted images are used as the photo-state electronic evidence of the video evidence. The text state electronic evidence can be text for explaining the picture state electronic evidence or text files such as contracts and forms corresponding to the picture state electronic evidence, and the text state electronic evidence can also be a text recognition result of a text picture.
And step S102, updating the text features and the picture features to obtain updated text features and updated picture features.
Specifically, the step S102 may include the following first and second parts:
and calculating the correlation among all the constituent elements in the text feature based on the attention mechanism, and updating the text feature by using the correlation to obtain the updated text feature.
It should be noted that, the relevance in the first portion refers to relevance between each constituent element in the text feature, where the text feature may be represented as a text feature matrix, and the constituent element may refer to a feature word vector. Alternatively, the above-mentioned process may include:
generating a query matrix Q of the text feature matrix based on an attention mechanism E Key matrix K E And value matrix V E The formula softmax (Q E ·K E T / d 1/2 ) Calculating the correlation softmax (), applying the formula softmax (). V between feature word vectors in the text feature matrix E The updated text feature E1, i.e. e1=attention (Q E ,K E ,V E ）=softmax(Q E ,K E )·V E Where Attention () is the Attention function, parameter d is the scale factor that prevents the molecular dot product value from being too large, and the value of parameter d is the dimension of the input feature.
And the second part calculates the correlation among all the constituent elements in the picture features based on the attention mechanism, and updates the picture features by utilizing the correlation to obtain updated picture features.
It should be noted that, the correlation in the second portion refers to a correlation between each component element in the picture feature, where the picture feature may be represented as a picture feature matrix R, and the component element may refer to a picture region feature vector. Optionally, the updated picture feature r1=attention (Q R ,K R ,V R ）=softmax(Q R ,K R )·V R Wherein parameter Q R 、K R 、V R The picture features generated based on the attention mechanismThe process of updating the picture features may be described above with reference to the query matrix, key matrix, and value matrix of the sign matrix R.
And step S103, fusing the updated text features and the updated picture features to obtain fusion features.
And step S104, calibrating the correlation between the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
According to the electronic evidence correlation calibration method, firstly, feature extraction is carried out on electronic evidence of different modes, then the extracted text features and picture features are updated based on the attention mechanism, in the process, correlation among all the constituent elements in the mode features is considered, the features are updated by utilizing the correlation, compared with the features directly extracted, the updated features can reflect the true meaning of the electronic evidence, so that more accurate electronic evidence correlation can be calibrated later, finally, the correlation of the electronic evidence of different modes is calibrated according to the fusion features fused by the updated text features and the updated picture features, the correlation calibration task of the multi-mode electronic evidence is realized, and the accuracy of the calibrated electronic evidence correlation is higher.
In some embodiments provided herein, the method for calibrating the correlation of electronic evidence provided herein may be implemented through a pre-trained multimodal heterographing attention network.
The multi-modal heterograph attention network may be a model trained by using text state training data and picture state training data labeled with corresponding correlations.
Fig. 2 is a schematic structural diagram of a multi-modal heterograph attention network according to an embodiment of the present application. In a possible implementation manner, as shown in connection with fig. 2, the multi-modal heterograph attention network may include a feature extraction module a, a multi-modal feature fusion module b, and a relevance calibration module c, where the feature extraction module a may include a text feature extraction module a1 and a picture feature extraction module a2, and the multi-modal feature fusion module b may include an in-mold attention module b1 and a fusion module b2.
The following describes each component module of the multi-mode heterograph attention network in sequence.
The text feature extraction module a1 may be configured to extract text features of the textual state electronic evidence. The text feature extraction module a1 can be used for capturing feature relations among words in the text state electronic evidence to obtain text features capable of representing the meaning of the text state electronic evidence.
Optionally, the text feature extraction process may include the following steps a-B:
and step A, acquiring word vector representation of the text state electronic evidence.
And B, extracting text feature relations among words in the text state electronic evidence based on the word vector representation, and generating text features by the text feature relations.
The step a may be implemented by using a pre-trained language model BERT, and the step B may be implemented by using a Bi-gating loop unit Bi-GRU, specifically, by using BERT to obtain semantic representation of textual electronic evidence, obtain a corresponding word vector initialization representation, and then using Bi-GRU to perform feature extraction on the word vector initialization representation to obtain text features.
In addition, the steps a-B may be implemented by using BERT, and it should be noted that, by using BERT, the dependency relationship between words may be extracted.
The picture feature extraction module a2 may be configured to extract picture features of the picture state electronic evidence.
It should be noted that, for example, for a feature extraction result of a picture, a high-level feature object is generally composed of a middle-level feature texture, the middle-level feature texture is generally composed of a low-level feature line, features of different levels are arranged in a sequence from low to high according to the levels, a set of sequence data can be obtained, a Recurrent Neural Network (RNN) generally takes the sequence data as input, an internal network of the RNN captures a feature relationship between sequences, and then the features are output in a sequence form.
Based on the foregoing, in one possible implementation manner, the picture feature extraction module a2 may include a cyclic neural network and a preset number of convolutional neural networks connected in series.
And the series preset number of convolutional neural networks are used for carrying out multi-level feature extraction on the picture state electronic evidence to obtain picture feature extraction results of different levels.
The cyclic neural network is used for extracting picture feature relations among picture feature extraction results of all levels, and picture features are generated by the picture feature relations.
Compared with a simple VGG network or a ResNet network, the picture feature extraction module a2 with the structure considers the feature relation among picture feature extraction results of different layers, and can better realize the picture feature extraction task.
Fig. 3 is a schematic structural diagram of a picture feature extraction module provided in an embodiment of the present application, and in combination with the schematic structural diagram shown in fig. 3, the picture feature extraction module a2 may include 5 convolutional neural networks a2_1 to a2_5 and a cyclic neural network a2_6 connected in series, the convolutional neural networks a2_1 to a2_5 sequentially perform feature extraction of different levels on picture state electronic evidences, respective picture feature extraction results are input to the cyclic neural network a2_6, and modeling is performed on sequence dependencies among the picture feature extraction results of different levels through the cyclic neural network a2_6, so as to obtain picture features, where the picture features may be represented as a picture feature matrix R. Optionally, the convolutional neural networks a2_1 to a2_5 may be 5 VGG19 networks with the same structure.
The in-mold attention module b1 may be used to update the text feature and the picture feature, and the update process may include: and for each feature in the text feature and the picture feature, calculating the correlation among all the constituent elements in the feature based on an attention mechanism, and updating the feature by utilizing the correlation to obtain an updated text feature and an updated picture feature. Other expressions may refer to step S102 described above.
The fusion module b2 may be configured to fuse the updated text feature and the updated picture feature to obtain a fusion feature.
In a possible implementation manner, the process of fusing the updated text feature and the updated picture feature by the fusing module b2 to obtain a fused feature may include the following steps C-D:
and C, respectively carrying out pooling treatment on the updated text features and the updated picture features to obtain pooled text features and pooled picture features with the same dimension.
Alternatively, the pooling process may be a maximum pooling process, and exemplary, the pooled text feature E1 '=maxpool (E1), the pooled picture feature R1' =maxpool (R1), where the parameter E1 is an updated text feature, and the parameter R1 is an updated picture feature.
And D, splicing the pooled text features and the pooled picture features to obtain fusion features.
And the correlation calibration module c can be used for calibrating the correlation between the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
Specifically, the process of calibrating the correlation may include:
and carrying out one-dimensional processing on the fusion features to obtain one-dimensional fusion features, and carrying out correlation calibration on the one-dimensional fusion features by utilizing a full-connection layer FC with an activation function of softmax to obtain the correlation of the textual state electronic evidence and the picture state electronic evidence.
The one-dimensional fusion characteristics can be projected to the two-class target space through the full-connection layer FC, and the electronic evidence correlation is obtained.
In particular, under the condition that a plurality of text state electronic evidences and a plurality of picture state electronic evidences exist in a group of electronic evidences, an electronic evidence pair can be formed by one text state electronic evidence and one picture state electronic evidence, then the application scheme is applied to determine the relativity of each text state electronic evidence and each picture state electronic evidence, the electronic evidence pair with relativity larger than a preset true threshold value is marked as a real evidence pair, the electronic evidence pair with relativity smaller than a preset false threshold value is marked as a false evidence pair, and the text state electronic evidence and the picture state electronic evidence in the group of electronic evidences are determined to be related according to the size relation of the real evidence pair and the false evidence pair under the condition that the number of the real evidence pairs is large, and the group of electronic evidences are real.
Fig. 4 is a flow chart illustrating another method for labeling relevance of electronic evidence according to an embodiment of the present application. As shown in connection with fig. 4, the method may comprise the steps of:
step S201 corresponds to step S101 described above, and will not be described in detail herein.
And step S202, weighting the text features and the picture features to obtain weighted text features and weighted picture features.
Specifically, the step S202 may include the following steps F-G:
and F, calculating a first correlation weight of the text feature and the picture feature based on an attention mechanism, and weighting the picture feature by using the first correlation weight to obtain a weighted text feature.
And G, calculating a second correlation weight of the picture feature and the text feature based on an attention mechanism, and weighting the text feature by using the second correlation weight to obtain a weighted picture feature.
In particular, the text feature may be represented as a text feature matrix E, the picture feature may be represented as a picture feature matrix R, and in one possible implementation, the first relevance weight may be represented as a softmax (Q E , K R ) Weighted text feature e0=attention (Q E , K R , V R ）= softmax(Q E , K R ) ·V R The second correlation weight may be expressed as softmax (Q R , K E ) Weighted picture feature r0=attention (Q R , K E , V E ）= softmax(Q R , K E )·V E Wherein the parameter Q E 、K E 、V E Respectively generating a query matrix, a key matrix and a value matrix of the text feature matrix E based on an attention mechanism, and parameters Q R 、K R 、V R The process of weighting the text feature and the picture feature may be described above for the query matrix, the key matrix, and the value matrix of the picture feature matrix R generated based on the attention mechanism, respectively.
And step 203, updating the weighted text features and the weighted picture features to obtain updated text features and updated picture features.
Note that, the method of feature update adopted in step S203 is identical to that of step S102 described above, and the difference between the two steps is that the object of feature update in step S203 is a weighted text feature and a weighted picture feature, and the object of feature update in step S102 is a text feature and a picture feature extracted in step S101.
Steps S204 to S205 are identical to steps S103 to S104 described above, and will not be described here.
According to the electronic evidence correlation calibration method, correlation among electronic evidence of different modes is considered, and the weighted text features and weighted picture features used for representing the characteristic relationship among the modes are updated by capturing the characteristic relationship inside the electronic evidence of the same mode, so that more accurate electronic evidence correlation can be calibrated later.
Fig. 5 is a schematic structural diagram of another multi-mode heterograph attention network according to the embodiment of the present application, and the electronic evidence correlation calibration method described above may be implemented by using the network shown in fig. 5.
As shown in fig. 5, the network may include a feature extraction module a, a multi-mode feature fusion module b, and a relevance calibration module c, where the feature extraction module a may include a text feature extraction module a1 and a picture feature extraction module a2, and the multi-mode feature fusion module b may include an inter-mode attention module b0, an in-mode attention module b1, and a fusion module b2.
Specifically, the text feature extraction module a1 performs feature extraction on the text state electronic evidence to obtain a text feature E, the picture feature extraction module a2 performs feature extraction on the picture state electronic evidence to obtain a picture feature R, the intermodule attention module b0 weights the text feature E and the picture feature R to obtain a weighted text feature E0 and a weighted picture feature R0, the in-module attention module b1 updates the weighted text feature E0 and the weighted picture feature R0 to obtain an updated text feature E1 and an updated picture feature R1, the fusion module b2 fuses the updated text feature E1 and the updated picture feature R1 to obtain a fusion feature V, and the correlation module c calibrates the correlation of the text state electronic evidence and the picture state electronic evidence according to the fusion feature V.
It should be noted that the process of weighting the text feature and the picture feature by the inter-module attention module b0 may include the following steps H-I:
and step H, calculating a first correlation weight of the text feature and the picture feature based on an attention mechanism, and weighting the picture feature by using the first correlation weight to obtain a weighted text feature.
And step I, calculating a second correlation weight of the picture feature and the text feature based on an attention mechanism, and weighting the text feature by using the second correlation weight to obtain a weighted picture feature.
The description of other modules may refer to the above description, and will not be repeated here.
The description of the electronic evidence correlation calibration device provided in the embodiment of the present application is provided below, and the electronic evidence correlation calibration device described below and the electronic evidence correlation calibration method described above may be referred to correspondingly.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an electronic evidence correlation calibration device according to an embodiment of the present application.
As shown in fig. 6, the apparatus may include:
a text feature extraction module 11, configured to extract text features of the textual electronic evidence;
a picture feature extraction module 12, configured to extract picture features of the picture state electronic evidence;
an in-mold attention module 13, configured to calculate, for each of the text feature and the picture feature, a correlation between each component element in the feature based on an attention mechanism, update the feature using the correlation, and obtain an updated text feature and an updated picture feature;
a fusion module 14, configured to fuse the updated text feature and the updated picture feature to obtain a fusion feature;
and the correlation calibration module 15 is used for calibrating the correlation between the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
Optionally, each module of the electronic evidence correlation calibration device is each functional module forming a pre-trained multi-mode different-composition attention network, the multi-mode different-composition attention network may be a model obtained by training with text state training data and picture state training data marked with corresponding correlations, and each module of the electronic evidence correlation calibration device and each module of the multi-mode different-composition attention network may be mutually referred to and correspond.
Optionally, the text feature extraction module may include a word vector representation acquisition module and a text feature generation module;
the word vector representation acquisition module is used for acquiring word vector representations of the text state electronic evidence;
the text feature generation module is used for extracting text feature relations among words in the text state electronic evidence based on the word vector representation, and generating text features according to the text feature relations.
Optionally, the image feature extraction module may include a cyclic neural network and a preset number of convolutional neural networks connected in series;
the series preset number of convolutional neural networks are used for carrying out multi-level feature extraction on the picture state electronic evidence to obtain picture feature extraction results of different levels;
the cyclic neural network is used for extracting picture feature relations among picture feature extraction results of all levels, and picture features are generated by the picture feature relations.
Optionally, the fusion module may include a pooling module and a stitching module;
the pooling module is used for pooling the updated text features and the updated picture features respectively to obtain pooled text features and pooled picture features with the same dimension;
and the splicing module is used for splicing the pooled text features and the pooled picture features to obtain fusion features.
Optionally, the device may further include an inter-module attention module, where the inter-module attention module is connected in series between a feature extraction module and the in-module attention module, and the feature extraction module is composed of the text feature extraction module and the picture feature extraction module;
the intermodule for intermodule attention is used for calculating a first correlation weight of the text feature and the picture feature based on an attention mechanism, weighting the picture feature by using the first correlation weight to obtain a weighted text feature, calculating a second correlation weight of the picture feature and the text feature based on an attention mechanism, weighting the text feature by using the second correlation weight to obtain a weighted picture feature, and updating the weighted text feature and the weighted picture feature by the intermodule attention module.
The electronic evidence correlation calibration device provided by the embodiment of the application can be applied to electronic evidence correlation calibration equipment, such as a terminal: cell phones, computers, etc. Optionally, fig. 7 shows a block diagram of a hardware structure of the electronic proof relevance calibration device, and referring to fig. 7, the hardware structure of the electronic proof relevance calibration device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;
in the embodiment of the application, the number of the processor 1, the communication interface 2, the memory 3 and the communication bus 4 is at least one, and the processor 1, the communication interface 2 and the memory 3 complete communication with each other through the communication bus 4;
the memory 3 may comprise a high-speed RAM memory, and may further comprise a non-volatile memory (non-volatile memory) or the like, such as at least one magnetic disk memory;
wherein the memory stores a program, the processor is operable to invoke the program stored in the memory, the program operable to:
extracting text features of the text state electronic evidence and extracting picture features of the picture state electronic evidence;
for each feature in the text feature and the picture feature, calculating the correlation among all constituent elements in the feature based on an attention mechanism, and updating the feature by utilizing the correlation to obtain an updated text feature and an updated picture feature;
fusing the updated text features and the updated picture features to obtain fused features;
and calibrating the correlation of the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
Alternatively, the refinement function and the extension function of the program may be described with reference to the above.
The embodiment of the application also provides a storage medium, which may store a program adapted to be executed by a processor, the program being configured to:
extracting text features of the text state electronic evidence and extracting picture features of the picture state electronic evidence;
for each feature in the text feature and the picture feature, calculating the correlation among all constituent elements in the feature based on an attention mechanism, and updating the feature by utilizing the correlation to obtain an updated text feature and an updated picture feature;
fusing the updated text features and the updated picture features to obtain fused features;
and calibrating the correlation of the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
Alternatively, the refinement function and the extension function of the program may be described with reference to the above.
In practical application, the correlation calibration can be performed on the electronic evidence to be stored before the multi-mode electronic evidence is stored, the authenticity of the electronic evidence to be stored is determined based on the calibrated correlation, the electronic evidence to be stored is considered to be true when the correlation is larger than a preset true threshold, the electronic evidence to be stored is allowed to be stored, and the electronic evidence to be stored is considered to be false when the correlation is smaller than the preset false threshold, and the storage is reminded or refused, so that the credibility of the stored electronic evidence is improved.
For example, the above-mentioned electronic evidence correlation calibration scheme can be applied to the production and operation activities such as electric power engineering infrastructure and line equipment operation inspection, and in the production and operation activities, the problems of limited evidence obtaining means, low evidence effectiveness, insufficient convenience in traditional judicial identification service and the like exist, and by constructing a corresponding electronic evidence management system, the trusted storage of the whole life cycle such as electronic evidence collection, solidification, transmission, sealing, inspection and identification can be realized, so that the public confidence of the electronic evidence is enhanced, the evidence obtaining efficiency and management and control capability are improved, and meanwhile, the tasks such as evidence authenticity, compliance identification, trusted tracking and monitoring can be realized by combining the characteristics of non-tamper property, transparency, safety and the like of a blockchain.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In the present specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and may be combined according to needs, and the same similar parts may be referred to each other.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (15)
1. The method for calibrating the correlation of the electronic evidence is characterized by comprising the following steps of:
extracting text features of the text state electronic evidence and extracting picture features of the picture state electronic evidence;
for each feature in the text feature and the picture feature, calculating the correlation among all constituent elements in the feature based on an attention mechanism, and updating the feature by utilizing the correlation to obtain an updated text feature and an updated picture feature;
fusing the updated text features and the updated picture features to obtain fused features;
and calibrating the correlation of the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
2. The method according to claim 1, wherein the electronic evidence relevance calibration method is implemented by a pre-trained multimodal heterographing attention network, which is a model trained with textual and pictorial training data labeled with corresponding relevance.
3. The method of claim 2, wherein the multimodal heterocomposition attention network comprises a text feature extraction module, a picture feature extraction module, an in-mold attention module, a fusion module, and a relevance calibration module;
the text feature extraction module is used for extracting text features of the text state electronic evidence;
the picture feature extraction module is used for extracting picture features of the picture state electronic evidence;
the in-mold attention module is used for calculating correlation among all constituent elements in the characteristics based on an attention mechanism for each characteristic in the text characteristics and the picture characteristics, and updating the characteristics by utilizing the correlation to obtain updated text characteristics and updated picture characteristics;
the fusion module is used for fusing the updated text characteristics and the updated picture characteristics to obtain fusion characteristics;
and the correlation calibration module is used for calibrating the correlation between the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
4. A method according to claim 3, wherein the process of extracting text features of the textual electronic proof by the text feature extraction module comprises:
acquiring word vector representation of the text state electronic evidence;
and extracting text feature relations among words in the text state electronic evidence based on the word vector representation, and generating text features according to the text feature relations.
5. A method according to claim 3, wherein the picture feature extraction module comprises a recurrent neural network and a preset number of convolutional neural networks in series;
the series preset number of convolutional neural networks are used for carrying out multi-level feature extraction on the picture state electronic evidence to obtain picture feature extraction results of different levels;
the cyclic neural network is used for extracting picture feature relations among picture feature extraction results of all levels, and picture features are generated by the picture feature relations.
6. A method according to claim 3, wherein the fusing module fuses the updated text feature and the updated picture feature to obtain a fused feature, comprising:
respectively carrying out pooling treatment on the updated text features and the updated picture features to obtain pooled text features and pooled picture features with the same dimension;
and splicing the pooled text features and the pooled picture features to obtain fusion features.
7. The method of any of claims 3-6, wherein the multimodal iso-composition attention network further comprises an inter-module attention module connected in series between a feature extraction module and the in-module attention module, the feature extraction module being comprised of the text feature extraction module and the picture feature extraction module;
the intermodule for intermodule attention is used for calculating a first correlation weight of the text feature and the picture feature based on an attention mechanism, weighting the picture feature by using the first correlation weight to obtain a weighted text feature, calculating a second correlation weight of the picture feature and the text feature based on an attention mechanism, weighting the text feature by using the second correlation weight to obtain a weighted picture feature, and updating the weighted text feature and the weighted picture feature by the intermodule attention module.
8. An electronic evidence correlation calibration device, comprising:
the text feature extraction module is used for extracting text features of the text state electronic evidence;
the picture feature extraction module is used for extracting picture features of the picture state electronic evidence;
the in-mold attention module is used for calculating the correlation among all the constituent elements in the characteristics based on an attention mechanism for each characteristic in the text characteristics and the picture characteristics, and updating the characteristics by utilizing the correlation to obtain updated text characteristics and updated picture characteristics;
the fusion module is used for fusing the updated text characteristics and the updated picture characteristics to obtain fusion characteristics;
and the correlation calibration module is used for calibrating the correlation between the text state electronic evidence and the picture state electronic evidence according to the fusion characteristics.
9. The apparatus of claim 8, wherein the modules of the apparatus are functional modules that form a pre-trained multimodal heterograph attention network that is a model trained using textual and pictorial training data labeled with corresponding correlations.
10. The apparatus of claim 9, wherein the text feature extraction module comprises a word vector representation acquisition module and a text feature generation module;
the word vector representation acquisition module is used for acquiring word vector representations of the text state electronic evidence;
the text feature generation module is used for extracting text feature relations among words in the text state electronic evidence based on the word vector representation, and generating text features according to the text feature relations.
11. The apparatus of claim 9, wherein the picture feature extraction module comprises a recurrent neural network and a predetermined number of convolutional neural networks in series;
the series preset number of convolutional neural networks are used for carrying out multi-level feature extraction on the picture state electronic evidence to obtain picture feature extraction results of different levels;
the cyclic neural network is used for extracting picture feature relations among picture feature extraction results of all levels, and picture features are generated by the picture feature relations.
12. The apparatus of claim 9, wherein the fusion module comprises a pooling module and a stitching module;
the pooling module is used for pooling the updated text features and the updated picture features respectively to obtain pooled text features and pooled picture features with the same dimension;
and the splicing module is used for splicing the pooled text features and the pooled picture features to obtain fusion features.
13. The apparatus according to any one of claims 8-12, further comprising an inter-module attention module connected in series between a feature extraction module and the in-module attention module, the feature extraction module being comprised of the text feature extraction module and the picture feature extraction module;
the intermodule for intermodule attention is used for calculating a first correlation weight of the text feature and the picture feature based on an attention mechanism, weighting the picture feature by using the first correlation weight to obtain a weighted text feature, calculating a second correlation weight of the picture feature and the text feature based on an attention mechanism, weighting the text feature by using the second correlation weight to obtain a weighted picture feature, and updating the weighted text feature and the weighted picture feature by the intermodule attention module.
14. An electronic evidence correlation calibration device, comprising: a memory and a processor;
the memory is used for storing programs;
the processor is configured to execute the program to implement the steps of the method for calibrating correlation of electronic evidence according to any one of claims 1-7.
15. A storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method for calibrating electronic evidence relevance according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310332705.2A CN116052171A (en) | 2023-03-31 | 2023-03-31 | Electronic evidence correlation calibration method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310332705.2A CN116052171A (en) | 2023-03-31 | 2023-03-31 | Electronic evidence correlation calibration method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116052171A true CN116052171A (en) | 2023-05-02 |
Family
ID=86122145
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310332705.2A Pending CN116052171A (en) | 2023-03-31 | 2023-03-31 | Electronic evidence correlation calibration method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116052171A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114332575A (en) * | 2021-12-28 | 2022-04-12 | 中电金信软件有限公司 | Multi-modal feature fusion method and device, electronic equipment and readable storage medium |
| CN114398961A (en) * | 2021-12-28 | 2022-04-26 | 西南交通大学 | Visual question-answering method based on multi-mode depth feature fusion and model thereof |
| CN114756763A (en) * | 2022-02-23 | 2022-07-15 | 北方工业大学 | False news detection method and device for social network |
| CN114925703A (en) * | 2022-06-14 | 2022-08-19 | 齐鲁工业大学 | Visual question-answering method and system with multi-granularity text representation and image-text fusion |
| CN115100664A (en) * | 2022-06-20 | 2022-09-23 | 济南大学 | Multi-mode false news identification method and system based on correlation information expansion |
| CN115775349A (en) * | 2022-12-09 | 2023-03-10 | 浙江微衡科技有限责任公司 | False news detection method and device based on multi-mode fusion |
-
2023
- 2023-03-31 CN CN202310332705.2A patent/CN116052171A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114332575A (en) * | 2021-12-28 | 2022-04-12 | 中电金信软件有限公司 | Multi-modal feature fusion method and device, electronic equipment and readable storage medium |
| CN114398961A (en) * | 2021-12-28 | 2022-04-26 | 西南交通大学 | Visual question-answering method based on multi-mode depth feature fusion and model thereof |
| CN114756763A (en) * | 2022-02-23 | 2022-07-15 | 北方工业大学 | False news detection method and device for social network |
| CN114925703A (en) * | 2022-06-14 | 2022-08-19 | 齐鲁工业大学 | Visual question-answering method and system with multi-granularity text representation and image-text fusion |
| CN115100664A (en) * | 2022-06-20 | 2022-09-23 | 济南大学 | Multi-mode false news identification method and system based on correlation information expansion |
| CN115775349A (en) * | 2022-12-09 | 2023-03-10 | 浙江微衡科技有限责任公司 | False news detection method and device based on multi-mode fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7213358B2 (en) | Identity verification method, identity verification device, computer equipment, and computer program | |
| CN109872242B (en) | Information pushing method and device | |
| CN111275107A (en) | Multi-label scene image classification method and device based on transfer learning | |
| CN112380435A (en) | Literature recommendation method and recommendation system based on heterogeneous graph neural network | |
| CN114419509B (en) | Multi-mode emotion analysis method and device and electronic equipment | |
| CN113011387B (en) | Network training and human face living body detection method, device, equipment and storage medium | |
| CN113515942A (en) | Text processing method and device, computer equipment and storage medium | |
| CN111444346B (en) | Word vector confrontation sample generation method and device for text classification | |
| CN111461175B (en) | Label recommendation model construction method and device of self-attention and cooperative attention mechanism | |
| CN114580794B (en) | Data processing method, apparatus, program product, computer device and medium | |
| CN112149604A (en) | Training method of video feature extraction model, video recommendation method and device | |
| CN114398973B (en) | Media content tag identification method, device, equipment and storage medium | |
| CN112819073A (en) | Classification network training method, image classification device and electronic equipment | |
| CN110097004B (en) | Facial expression recognition method and device | |
| CN113688814B (en) | Image recognition method and device | |
| CN111159464A (en) | Audio clip detection method and related equipment | |
| CN112765481B (en) | Data processing method, device, computer and readable storage medium | |
| CN116578738B (en) | Graph-text retrieval method and device based on graph attention and generating countermeasure network | |
| CN117037182A (en) | Image authenticity detection method, device, computer equipment and storage medium | |
| CN116932770A (en) | Multi-mode knowledge graph entity alignment method and system based on interaction between modes | |
| CN116342978A (en) | Target detection network training and target detection method and device, and electronic equipment | |
| CN116052171A (en) | Electronic evidence correlation calibration method, device, equipment and storage medium | |
| CN113610228B (en) | Method and device for constructing neural network model | |
| CN117011909A (en) | Training method of face recognition model, face recognition method and device | |
| CN114462546A (en) | Feature distillation method, system, equipment and medium based on multi-model fusion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230502 |