CN112734205A - Model confidence degree analysis method and device, electronic equipment and computer storage medium - Google Patents
Model confidence degree analysis method and device, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN112734205A CN112734205A CN202011637424.0A CN202011637424A CN112734205A CN 112734205 A CN112734205 A CN 112734205A CN 202011637424 A CN202011637424 A CN 202011637424A CN 112734205 A CN112734205 A CN 112734205A
- Authority
- CN
- China
- Prior art keywords
- text
- result
- index
- standard
- conversion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
- G06F16/353—Clustering; Classification into predefined classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/151—Transformation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Human Resources & Organizations (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Data Mining & Analysis (AREA)
- Educational Administration (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Databases & Information Systems (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- Molecular Biology (AREA)
- Software Systems (AREA)
- Game Theory and Decision Science (AREA)
- Biophysics (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a performance test technology, and discloses a model confidence degree analysis method, which comprises the following steps: acquiring a standard text and a standard conversion result corresponding to the standard text; performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result; generating a prediction result index of the standard text according to the prediction conversion result; generating a comparison result index according to the prediction conversion result and the standard conversion result; and constructing a model confidence index by using the prediction result index and the comparison result index, and performing confidence analysis on the text conversion model by using the model confidence index. In addition, the invention also relates to a block chain technology, and the standard text and the standard conversion result can be stored in the nodes of the block chain. The invention also provides a model confidence coefficient analysis device, equipment and a storage medium. The method and the device can solve the problem that the accuracy of confidence evaluation on the output result of the model is low.
Description
Technical Field
The invention relates to the technical field of performance testing, in particular to a model confidence degree analysis method and device, electronic equipment and a computer readable storage medium.
Background
In the field of natural language processing, intelligent models are required to complete various tasks, such as information extraction, event classification, topic extraction, named entity identification, and the like. In order to improve the accuracy and other performances of the intelligent model, the confidence evaluation of the output result of the intelligent model is required.
Most of confidence evaluation methods of the existing intelligent models are confidence evaluation methods based on big data, namely, an overall output index is designed based on a large amount of intelligent model output so as to adjust parameters of the intelligent models according to the overall output index. However, in this method, since the index is designed based on a large amount of output data as a whole, it is not possible to evaluate the confidence level of a single intelligent model output, resulting in low accuracy of the confidence level evaluation of the intelligent model output.
Disclosure of Invention
The invention provides a method and a device for analyzing model confidence and a computer readable storage medium, and mainly aims to solve the problem that the accuracy of confidence evaluation on a model output result is low.
In order to achieve the above object, the present invention provides a model confidence analysis method, including:
acquiring a standard text and a standard conversion result corresponding to the standard text;
performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result;
generating a prediction result index of the standard text according to the prediction conversion result;
generating a comparison result index according to the prediction conversion result and the standard conversion result;
and constructing a model confidence index by using the prediction result index and the comparison result index, and performing confidence analysis on the text conversion model by using the model confidence index.
Optionally, the performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result includes:
performing convolution on the standard text by utilizing a convolution layer of the convolutional neural network to obtain a convolution text;
pooling the convolution texts to obtain feature texts;
carrying out full-connection processing on the feature text to obtain a full-connection text;
and carrying out probability classification on the full-connection text by using an activation function to obtain a prediction conversion result.
Optionally, the generating a prediction result indicator of the standard text according to the prediction conversion result includes:
carrying out statement structure labeling on the prediction conversion result to obtain a labeling result;
calculating according to the labeling result to obtain a text grammatical structure index;
performing word vector conversion on the prediction conversion result to obtain a result vector;
calculating the text semantic index by using the result vector;
and aggregating the text syntactic structure index and the text semantic index into a prediction result index.
Optionally, the performing word vector conversion on the prediction conversion result to obtain a result vector includes:
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
and carrying out word vector coding on the plurality of predicted text participles to obtain a result vector corresponding to each predicted text participle.
Optionally, the generating a comparison result indicator according to the prediction conversion result and the standard conversion result includes:
traversing the prediction conversion result and determining the prediction text length of the prediction conversion result;
traversing the standard conversion result and determining the standard text length of the standard conversion result;
calculating a text length index according to the predicted text length and the standard text length;
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
performing text word segmentation processing on the standard conversion result to obtain a plurality of standard text words;
calculating word frequency indexes according to the plurality of predicted text participles and the plurality of standard text participles;
and collecting the text length index and the word frequency index as a comparison result index.
Optionally, the constructing a model confidence indicator by using the predicted result indicator and the comparison result indicator includes:
acquiring a preset index weight coefficient;
and performing weighting operation on the prediction result index and the comparison result index by using a preset weight coefficient to obtain a model confidence coefficient index.
Optionally, the performing a weighting operation on the predicted result index and the comparison result index by using a preset weight coefficient to obtain a model confidence index includes:
carrying out weighting operation by using the following weighting algorithm to obtain a model confidence index:
Z＝δ*G+ε*H+μ*P+τ*Q
wherein Z is the confidence index of the model, G is the index of the grammatical structure of the text, H is the index of the semantic meaning of the text, P is the index of the length of the text, Q is the index of the word frequency, and delta, epsilon, mu and tau are preset index weight coefficients.
In order to solve the above problem, the present invention also provides a model confidence analyzing apparatus, including:
the text acquisition module is used for acquiring a standard text and a standard conversion result corresponding to the standard text;
the text conversion module is used for performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result;
the first index generation module is used for generating a prediction result index of the standard text according to the prediction conversion result;
the second index generation module is used for generating a comparison result index according to the prediction conversion result and the standard conversion result;
and the confidence coefficient analysis module is used for constructing a model confidence coefficient index by using the prediction result index and the comparison result index and carrying out confidence coefficient analysis on the text conversion model by using the model confidence coefficient index.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one instruction; and
and the processor executes the instructions stored in the memory to realize the model confidence analysis method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium having at least one instruction stored therein, where the at least one instruction is executed by a processor in an electronic device to implement the model confidence analysis method described above.
According to the embodiment of the invention, the text conversion is carried out on the obtained standard text, the prediction result index is generated according to the converted prediction conversion result, the result index is compared by using the prediction conversion result and the standard conversion result corresponding to the standard text, and the confidence index is constructed by using the prediction result index and the comparison result index to analyze the confidence coefficient of single output of the model, so that the overall confidence index is prevented from being designed based on massive model output, and the accuracy of performing confidence evaluation on the model output result is improved. Therefore, the model confidence degree analysis method, the model confidence degree analysis device, the electronic equipment and the computer readable storage medium can solve the problem that the accuracy of the confidence degree evaluation of the model output result is not high.
Drawings
FIG. 1 is a schematic flow chart illustrating a model confidence analysis method according to an embodiment of the present invention;
FIG. 2 is a functional block diagram of a model confidence analysis apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device for implementing the model confidence analysis method according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The embodiment of the application provides a model confidence coefficient analysis method. The execution subject of the model confidence analysis method includes, but is not limited to, at least one of electronic devices that can be configured to execute the method provided by the embodiments of the present application, such as a server, a terminal, and the like. In other words, the model confidence analysis method may be performed by software or hardware installed in the terminal device or the server device, and the software may be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Fig. 1 is a schematic flow chart of a model confidence analysis method according to an embodiment of the present invention. In this embodiment, the model confidence analysis method includes:
and S1, acquiring the standard text and the standard conversion result corresponding to the standard text.
In the embodiment of the present invention, the standard text may be any text, such as a report text of current news, a product introduction text, an activity plan text, and the like.
In detail, the standard conversion result corresponding to the standard text includes any result output after the standard text is analyzed and processed, for example, when a text compression task is executed, the standard conversion result corresponding to the standard text is the standard text after the text compression.
According to the embodiment of the invention, the standard text and the standard conversion result can be obtained from the block chain node for storing the standard text and the standard conversion result by using the python statement with the data capture function, and the efficiency of obtaining the standard text and the standard conversion result can be improved by using the high throughput of the block chain to the data.
And S2, performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result.
In the embodiment of the present invention, the text conversion model includes, but is not limited to, a text compression model, a text translation model, and the like.
In detail, the text conversion model includes a convolutional layer, a pooling layer, and a fully-connected layer.
The convolution layer is used for performing convolution processing on the text, firstly locally perceiving each feature in the text, and then performing comprehensive operation on the local feature at a higher level so as to obtain global information;
the pooling layer is used for pooling the text after convolution for feature dimension reduction, so that the quantity of data and parameters can be reduced, and the fault tolerance of the model can be improved;
and the full connection layer is used for performing linear classification on the output result of the pooling layer by utilizing an activation function, and is particularly used for performing linear combination on the extracted high-level feature vector and outputting the final text conversion result.
In an embodiment of the present invention, the text conversion model is a convolutional neural network, and performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result includes:
performing convolution on the standard text by utilizing a convolution layer of the convolutional neural network to obtain a convolution text;
pooling the convolution texts to obtain feature texts;
carrying out full-connection processing on the feature text to obtain a full-connection text;
and carrying out probability classification on the full-connection text by using an activation function to obtain a prediction conversion result.
In detail, the convolving the standard image with the convolution layer of the text conversion model includes multiplying the standard text with a preset convolution kernel matrix. The activation function includes but is not limited to softmax activation function, sigmoid activation function.
Specifically, the predicted conversion result includes any result output after the standard text is subjected to the text conversion process by the text conversion model, for example, there is a standard text "this is a white gardenia", when the text conversion model is a text compression model, the standard text is subjected to text conversion by the text compression model, and the obtained text conversion result is "this is a flower".
And S3, generating a prediction result index of the standard text according to the prediction conversion result.
In the embodiment of the present invention, the prediction result indicator includes, but is not limited to, a text semantic indicator and a text syntactic indicator.
In detail, the generating a prediction result index of the standard text according to the prediction conversion result includes:
carrying out statement structure labeling on the prediction conversion result to obtain a labeling result;
calculating according to the labeling result to obtain a text grammatical structure index;
performing word vector conversion on the prediction conversion result to obtain a result vector;
calculating the text semantic index by using the result vector;
and aggregating the text syntactic structure index and the text semantic index into a prediction result index.
According to the embodiment of the invention, a sentence structure labeling is carried out on the predictive conversion result by using an HMM (Hidden Markov model), wherein the sentence structure labeling can label grammatical structures such as a subject, a predicate and an object in the predictive conversion result, and when the grammatical structure of the predictive conversion result is more complete, the higher the confidence of the predictive conversion result is.
In detail, the calculating according to the labeling result to obtain the text grammar structure index includes:
and calculating a text grammar structure index according to the labeling result by using the following formula:
G＝α+β+γ
wherein G is the text grammar structure index, α, β, and γ are variable indexes, α is 1 when a subject is present in the predictive conversion result, α is 0 when the subject is not present in the predictive conversion result, β is 1 when a predicate is present in the predictive conversion result, β is 0 when the predicate is not present in the predictive conversion result, γ is 1 when an object is present in the predictive conversion result, and γ is 0 when the object is not present in the predictive conversion result.
In the embodiment of the present invention, the text semantic index refers to how much the semantic content of the predictive conversion result is, and when the numerical value of the text syntactic structure index is larger, it indicates that the syntactic structure of the predictive conversion result is more complete, and the confidence of the predictive conversion result is higher.
In detail, the performing word vector transformation on the prediction transformation result to obtain a result vector includes:
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
and carrying out word vector coding on the plurality of predicted text participles to obtain a result vector corresponding to each predicted text participle.
Specifically, word segmentation is performed on the prediction conversion result by using a preset standard word bank to obtain a plurality of predicted text words, wherein the preset standard word bank contains a plurality of standard words, such as "eat" and "sleep", and the preset standard word bank can be used for performing text word segmentation on the prediction conversion result.
The embodiment of the invention carries out word segmentation processing on the standard text, can divide the prediction conversion result with larger length into prediction text words, and has higher processing efficiency and accuracy by analyzing and processing a plurality of words compared with directly carrying out processing through the prediction conversion result.
In detail, the embodiment of the invention uses a preset coding model to perform word vector coding on the predicted text participle to obtain a result vector, wherein the coding model includes but is not limited to a Character Glyph coding model and a One-hot coding model.
Further, the calculating the text semantic indicator by using the result vector includes:
calculating the text semantic index H by using the result vector according to the following information entropy algorithm:
wherein Y is a set of the plurality of predicted text participles, xiThe result vector of the ith predicted text word segmentation in Y, k is the number of the result vectors in Y,
is the frequency of occurrence of the ith result vector in Y.
And S4, generating a comparison result index according to the prediction conversion result and the standard conversion result.
In the embodiment of the present invention, the comparison result indicator includes, but is not limited to, a text length indicator and a word frequency indicator.
In detail, the generating a comparison result index according to the prediction conversion result and the standard conversion result includes:
traversing the prediction conversion result and determining the prediction text length of the prediction conversion result;
traversing the standard conversion result and determining the standard text length of the standard conversion result;
calculating a text length index according to the predicted text length and the standard text length;
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
performing text word segmentation processing on the standard conversion result to obtain a plurality of standard text words;
calculating word frequency indexes according to the plurality of predicted text participles and the plurality of standard text participles;
and collecting the text length index and the word frequency index as a comparison result index.
In an embodiment of the present invention, the calculating a text length index according to the predicted text length and the standard text length includes:
calculating a text length index by using the following length index algorithm:
wherein, P is the text length index, theta is the predicted text length, and rho is the standard text length.
In detail, the text length index refers to the length of the prediction conversion result relative to the standard text, and when the text conversion model is a text compression model, the smaller the text length index is, the higher the confidence of the prediction conversion result is; when the text conversion model is a text translation model, the larger the text length index is, the higher the confidence of the prediction conversion result is.
Further, the step of performing text segmentation on the standard conversion result is the same as the step of performing text segmentation on the prediction conversion result in step S3, and is not described herein again.
In detail, the word frequency indicator refers to the frequency of occurrence of the text participles included in the predicted conversion result in the standard conversion result, for example, if there is a predicted conversion result "which is a flower", and a standard conversion result "which is a flower", then the frequency of occurrence of the participles "which is a flower", "a flower", and "a flower" in the standard conversion result is 3 times. When the occurrence frequency of the text participles contained in the prediction conversion result in the standard conversion result indicates that the confidence of the prediction conversion result is higher.
S5, constructing a model confidence index by using the prediction result index and the comparison result index, and performing confidence analysis on the text conversion model by using the model confidence index.
In an embodiment of the present invention, the constructing a model confidence indicator by using the predicted result indicator and the comparison result indicator includes:
acquiring a preset index weight coefficient;
and performing weighting operation on the prediction result index and the comparison result index by using a preset weight coefficient to obtain a model confidence coefficient index.
In detail, the index weight coefficient may be given by a user person in advance.
Specifically, the performing a weighted operation on the predicted result index and the compared result index by using a preset weight coefficient to obtain a model confidence index includes:
carrying out weighting operation by using the following weighting algorithm to obtain a model confidence index:
Z＝δ*G+ε*H+μ*P+τ*Q
wherein Z is the confidence index of the model, G is the index of the grammatical structure of the text, H is the index of the semantic meaning of the text, P is the index of the length of the text, Q is the index of the word frequency, and delta, epsilon, mu and tau are preset index weight coefficients.
In an embodiment of the present invention, the confidence indicator may be used to analyze an accuracy of an output result of the text conversion model, and the confidence indicator is used to perform confidence analysis on the text conversion model, that is, the accuracy of the output result of the text conversion model is analyzed according to the magnitude of the confidence indicator, for example, when the value of the confidence indicator is larger, the accuracy of the output result of the text conversion model is higher, and when the value of the confidence indicator is smaller, the accuracy of the output result of the text conversion model is lower.
According to the embodiment of the invention, the text conversion is carried out on the obtained standard text, the prediction result index is generated according to the converted prediction conversion result, the result index is compared by using the prediction conversion result and the standard conversion result corresponding to the standard text, and the confidence index is constructed by using the prediction result index and the comparison result index to analyze the confidence coefficient of single output of the model, so that the overall confidence index is prevented from being designed based on massive model output, and the accuracy of performing confidence evaluation on the model output result is improved. Therefore, the model confidence coefficient analysis method provided by the invention can solve the problem of low accuracy of the confidence coefficient evaluation of the model output result.
Fig. 2 is a functional block diagram of a model confidence analysis apparatus according to an embodiment of the present invention.
The model confidence analysis apparatus 100 according to the present invention may be installed in an electronic device. According to the realized functions, the model confidence analysis apparatus 100 may include a text acquisition module 101, a text conversion module 102, a first index generation module 103, a second index generation module 104, and a confidence analysis module 105. The module of the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the text obtaining module 101 is configured to obtain a standard text and a standard conversion result corresponding to the standard text.
In the embodiment of the present invention, the standard text may be any text, such as a report text of current news, a product introduction text, an activity plan text, and the like.
In detail, the standard conversion result corresponding to the standard text includes any result output after the standard text is analyzed and processed, for example, when a text compression task is executed, the standard conversion result corresponding to the standard text is the standard text after the text compression.
According to the embodiment of the invention, the standard text and the standard conversion result can be obtained from the block chain node for storing the standard text and the standard conversion result by using the python statement with the data capture function, and the efficiency of obtaining the standard text and the standard conversion result can be improved by using the high throughput of the block chain to the data.
The text conversion module 102 is configured to perform text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result.
In the embodiment of the present invention, the text conversion model includes, but is not limited to, a text compression model, a text translation model, and the like.
In detail, the text conversion model includes a convolutional layer, a pooling layer, and a fully-connected layer.
The convolution layer is used for performing convolution processing on the text, firstly locally perceiving each feature in the text, and then performing comprehensive operation on the local feature at a higher level so as to obtain global information;
the pooling layer is used for pooling the text after convolution for feature dimension reduction, so that the quantity of data and parameters can be reduced, and the fault tolerance of the model can be improved;
and the full connection layer is used for performing linear classification on the output result of the pooling layer by utilizing an activation function, and is particularly used for performing linear combination on the extracted high-level feature vector and outputting the final text conversion result.
In an embodiment of the present invention, the text conversion model is a convolutional neural network, and the text conversion module 102 is specifically configured to:
performing convolution on the standard text by utilizing a convolution layer of the convolutional neural network to obtain a convolution text;
pooling the convolution texts to obtain feature texts;
carrying out full-connection processing on the feature text to obtain a full-connection text;
and carrying out probability classification on the full-connection text by using an activation function to obtain a prediction conversion result.
In detail, the convolving the standard image with the convolution layer of the text conversion model includes multiplying the standard text with a preset convolution kernel matrix. The activation function includes but is not limited to softmax activation function, sigmoid activation function.
Specifically, the predicted conversion result includes any result output after the standard text is subjected to the text conversion process by the text conversion model, for example, there is a standard text "this is a white gardenia", when the text conversion model is a text compression model, the standard text is subjected to text conversion by the text compression model, and the obtained text conversion result is "this is a flower".
The first index generating module 103 is configured to generate a prediction result index of the standard text according to the prediction conversion result.
In the embodiment of the present invention, the prediction result indicator includes, but is not limited to, a text semantic indicator and a text syntactic indicator.
In detail, the first index generating module 103 is specifically configured to:
carrying out statement structure labeling on the prediction conversion result to obtain a labeling result;
calculating according to the labeling result to obtain a text grammatical structure index;
performing word vector conversion on the prediction conversion result to obtain a result vector;
calculating the text semantic index by using the result vector;
and aggregating the text syntactic structure index and the text semantic index into a prediction result index.
According to the embodiment of the invention, a sentence structure labeling is carried out on the predictive conversion result by using an HMM (Hidden Markov model), wherein the sentence structure labeling can label grammatical structures such as a subject, a predicate and an object in the predictive conversion result, and when the grammatical structure of the predictive conversion result is more complete, the higher the confidence of the predictive conversion result is.
In detail, the calculating according to the labeling result to obtain the text grammar structure index includes:
and calculating a text grammar structure index according to the labeling result by using the following formula:
G＝α+β+γ
wherein G is the text grammar structure index, α, β, and γ are variable indexes, α is 1 when a subject is present in the predictive conversion result, α is 0 when the subject is not present in the predictive conversion result, β is 1 when a predicate is present in the predictive conversion result, β is 0 when the predicate is not present in the predictive conversion result, γ is 1 when an object is present in the predictive conversion result, and γ is 0 when the object is not present in the predictive conversion result.
In the embodiment of the present invention, the text semantic index refers to how much the semantic content of the predictive conversion result is, and when the numerical value of the text syntactic structure index is larger, it indicates that the syntactic structure of the predictive conversion result is more complete, and the confidence of the predictive conversion result is higher.
In detail, the performing word vector transformation on the prediction transformation result to obtain a result vector includes:
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
and carrying out word vector coding on the plurality of predicted text participles to obtain a result vector corresponding to each predicted text participle.
Specifically, word segmentation is performed on the prediction conversion result by using a preset standard word bank to obtain a plurality of predicted text words, wherein the preset standard word bank contains a plurality of standard words, such as "eat" and "sleep", and the preset standard word bank can be used for performing text word segmentation on the prediction conversion result.
The embodiment of the invention carries out word segmentation processing on the standard text, can divide the prediction conversion result with larger length into prediction text words, and has higher processing efficiency and accuracy by analyzing and processing a plurality of words compared with directly carrying out processing through the prediction conversion result.
In detail, the embodiment of the invention uses a preset coding model to perform word vector coding on the predicted text participle to obtain a result vector, wherein the coding model includes but is not limited to a Character Glyph coding model and a One-hot coding model.
Further, the calculating the text semantic indicator by using the result vector includes:
calculating the text semantic index H by using the result vector according to the following information entropy algorithm:
wherein Y is a set of the plurality of predicted text participles, xiThe result vector of the ith predicted text word segmentation in Y, k is the number of the result vectors in Y,
is the frequency of occurrence of the ith result vector in Y.
The second index generating module 104 is configured to generate a comparison result index according to the prediction conversion result and the standard conversion result.
In the embodiment of the present invention, the comparison result indicator includes, but is not limited to, a text length indicator and a word frequency indicator.
In detail, the second index generating module 104 is specifically configured to:
traversing the prediction conversion result and determining the prediction text length of the prediction conversion result;
traversing the standard conversion result and determining the standard text length of the standard conversion result;
calculating a text length index according to the predicted text length and the standard text length;
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
performing text word segmentation processing on the standard conversion result to obtain a plurality of standard text words;
calculating word frequency indexes according to the plurality of predicted text participles and the plurality of standard text participles;
and collecting the text length index and the word frequency index as a comparison result index.
In an embodiment of the present invention, the calculating a text length index according to the predicted text length and the standard text length includes:
calculating a text length index by using the following length index algorithm:
wherein, P is the text length index, theta is the predicted text length, and rho is the standard text length.
In detail, the text length index refers to the length of the prediction conversion result relative to the standard text, and when the text conversion model is a text compression model, the smaller the text length index is, the higher the confidence of the prediction conversion result is; when the text conversion model is a text translation model, the larger the text length index is, the higher the confidence of the prediction conversion result is.
Further, the step of performing text segmentation on the standard conversion result is the same as the step of performing text segmentation on the prediction conversion result in step S3, and is not described herein again.
In detail, the word frequency indicator refers to the frequency of occurrence of the text participles included in the predicted conversion result in the standard conversion result, for example, if there is a predicted conversion result "which is a flower", and a standard conversion result "which is a flower", then the frequency of occurrence of the participles "which is a flower", "a flower", and "a flower" in the standard conversion result is 3 times. When the occurrence frequency of the text participles contained in the prediction conversion result in the standard conversion result indicates that the confidence of the prediction conversion result is higher.
The confidence coefficient analysis module 105 is configured to construct a model confidence coefficient index by using the prediction result index and the comparison result index, and perform confidence coefficient analysis on the text conversion model by using the model confidence coefficient index.
In an embodiment of the present invention, the confidence level analyzing module 105 is specifically configured to:
acquiring a preset index weight coefficient;
and performing weighting operation on the prediction result index and the comparison result index by using a preset weight coefficient to obtain a model confidence coefficient index.
In detail, the index weight coefficient may be given by a user person in advance.
Specifically, the performing a weighted operation on the predicted result index and the compared result index by using a preset weight coefficient to obtain a model confidence index includes:
carrying out weighting operation by using the following weighting algorithm to obtain a model confidence index:
Z＝δ*G+ε*H+μ*P+τ*Q
wherein Z is the confidence index of the model, G is the index of the grammatical structure of the text, H is the index of the semantic meaning of the text, P is the index of the length of the text, Q is the index of the word frequency, and delta, epsilon, mu and tau are preset index weight coefficients.
In an embodiment of the present invention, the confidence indicator may be used to analyze an accuracy of an output result of the text conversion model, and the confidence indicator is used to perform confidence analysis on the text conversion model, that is, the accuracy of the output result of the text conversion model is analyzed according to the magnitude of the confidence indicator, for example, when the value of the confidence indicator is larger, the accuracy of the output result of the text conversion model is higher, and when the value of the confidence indicator is smaller, the accuracy of the output result of the text conversion model is lower.
According to the embodiment of the invention, the text conversion is carried out on the obtained standard text, the prediction result index is generated according to the converted prediction conversion result, the result index is compared by using the prediction conversion result and the standard conversion result corresponding to the standard text, and the confidence index is constructed by using the prediction result index and the comparison result index to analyze the confidence coefficient of single output of the model, so that the overall confidence index is prevented from being designed based on massive model output, and the accuracy of performing confidence evaluation on the model output result is improved. Therefore, the model confidence coefficient analysis device provided by the invention can solve the problem of low accuracy of confidence coefficient evaluation on the model output result.
Fig. 3 is a schematic structural diagram of an electronic device implementing a model confidence analysis method according to an embodiment of the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as a model confidence analysis program 12, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only to store application software installed in the electronic device 1 and various types of data, such as codes of the model confidence analysis program 12, but also to temporarily store data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (e.g., model confidence analysis programs, etc.) stored in the memory 11 and calling data stored in the memory 11.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 3 shows only an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices.
Optionally, the electronic device 1 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The model confidence analysis program 12 stored in the memory 11 of the electronic device 1 is a combination of instructions that, when executed in the processor 10, may implement:
acquiring a standard text and a standard conversion result corresponding to the standard text;
performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result;
generating a prediction result index of the standard text according to the prediction conversion result;
generating a comparison result index according to the prediction conversion result and the standard conversion result;
and constructing a model confidence index by using the prediction result index and the comparison result index, and performing confidence analysis on the text conversion model by using the model confidence index.
Specifically, the specific implementation method of the processor 10 for the instruction may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, which is not described herein again.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. The computer readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
The present invention also provides a computer-readable storage medium, storing a computer program which, when executed by a processor of an electronic device, may implement:
acquiring a standard text and a standard conversion result corresponding to the standard text;
performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result;
generating a prediction result index of the standard text according to the prediction conversion result;
generating a comparison result index according to the prediction conversion result and the standard conversion result;
and constructing a model confidence index by using the prediction result index and the comparison result index, and performing confidence analysis on the text conversion model by using the model confidence index.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A method of model confidence analysis, the method comprising:
acquiring a standard text and a standard conversion result corresponding to the standard text;
performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result;
generating a prediction result index of the standard text according to the prediction conversion result;
generating a comparison result index according to the prediction conversion result and the standard conversion result;
and constructing a model confidence index by using the prediction result index and the comparison result index, and performing confidence analysis on the text conversion model by using the model confidence index.
2. The method for model confidence analysis according to claim 1, wherein the text conversion of the standard text using a preset text conversion model to obtain a prediction conversion result comprises:
performing convolution on the standard text by utilizing a convolution layer of the convolutional neural network to obtain a convolution text;
pooling the convolution texts to obtain feature texts;
carrying out full-connection processing on the feature text to obtain a full-connection text;
and carrying out probability classification on the full-connection text by using an activation function to obtain a prediction conversion result.
3. The model confidence analysis method of claim 1, wherein the generating the prediction result indicator for the standard text from the predictive conversion result comprises:
carrying out statement structure labeling on the prediction conversion result to obtain a labeling result;
calculating according to the labeling result to obtain a text grammatical structure index;
performing word vector conversion on the prediction conversion result to obtain a result vector;
calculating the text semantic index by using the result vector;
and aggregating the text syntactic structure index and the text semantic index into a prediction result index.
4. The method for model confidence analysis according to claim 3, wherein the performing a word vector transformation on the result of the predictive transformation to obtain a result vector comprises:
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
and carrying out word vector coding on the plurality of predicted text participles to obtain a result vector corresponding to each predicted text participle.
5. The model confidence analysis method of claim 1, wherein the generating a comparison result indicator based on the predicted transformation result and the standard transformation result comprises:
traversing the prediction conversion result and determining the prediction text length of the prediction conversion result;
traversing the standard conversion result and determining the standard text length of the standard conversion result;
calculating a text length index according to the predicted text length and the standard text length;
performing text word segmentation processing on the prediction conversion result to obtain a plurality of prediction text words;
performing text word segmentation processing on the standard conversion result to obtain a plurality of standard text words;
calculating word frequency indexes according to the plurality of predicted text participles and the plurality of standard text participles;
and collecting the text length index and the word frequency index as a comparison result index.
6. The model confidence analysis method of any of claims 1 to 5, wherein the constructing a model confidence indicator using the predicted result indicator and the compared result indicator comprises:
acquiring a preset index weight coefficient;
and performing weighting operation on the prediction result index and the comparison result index by using a preset weight coefficient to obtain a model confidence coefficient index.
7. The method for model confidence analysis according to claim 6, wherein the weighting the predicted result indicator and the compared result indicator by using a preset weight coefficient to obtain the model confidence indicator comprises:
carrying out weighting operation by using the following weighting algorithm to obtain a model confidence index:
Z＝δ*G+ε*H+μ*P+τ*Q
wherein Z is the confidence index of the model, G is the index of the grammatical structure of the text, H is the index of the semantic meaning of the text, P is the index of the length of the text, Q is the index of the word frequency, and delta, epsilon, mu and tau are preset index weight coefficients.
8. An apparatus for model confidence analysis, the apparatus comprising:
the text acquisition module is used for acquiring a standard text and a standard conversion result corresponding to the standard text;
the text conversion module is used for performing text conversion on the standard text by using a preset text conversion model to obtain a prediction conversion result;
the first index generation module is used for generating a prediction result index of the standard text according to the prediction conversion result;
the second index generation module is used for generating a comparison result index according to the prediction conversion result and the standard conversion result;
and the confidence coefficient analysis module is used for constructing a model confidence coefficient index by using the prediction result index and the comparison result index and carrying out confidence coefficient analysis on the text conversion model by using the model confidence coefficient index.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the model confidence analysis method of any of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out a method for model confidence analysis according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637424.0A CN112734205A (en) | 2020-12-31 | 2020-12-31 | Model confidence degree analysis method and device, electronic equipment and computer storage medium |
| PCT/CN2021/083175 WO2022141838A1 (en) | 2020-12-31 | 2021-03-26 | Model confidence analysis method and apparatus, electronic device and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011637424.0A CN112734205A (en) | 2020-12-31 | 2020-12-31 | Model confidence degree analysis method and device, electronic equipment and computer storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112734205A true CN112734205A (en) | 2021-04-30 |
Family
ID=75608731
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011637424.0A Pending CN112734205A (en) | 2020-12-31 | 2020-12-31 | Model confidence degree analysis method and device, electronic equipment and computer storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112734205A (en) |
| WO (1) | WO2022141838A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113268895A (en) * | 2021-07-20 | 2021-08-17 | 湖南高至科技有限公司 | Combined simulation method, device, equipment and medium based on semantic mapping |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102449614B1 (en) * | 2015-11-06 | 2022-09-29 | 삼성전자주식회사 | Apparatus and method for evaluating machine translation quality using distributed representation, machine translation apparatus, and apparatus for constructing distributed representation model |
| CN110134971B (en) * | 2018-02-08 | 2022-12-16 | 腾讯科技（深圳）有限公司 | Method and device for machine translation and computer readable storage medium |
| CN109858042B (en) * | 2018-11-20 | 2024-02-20 | 科大讯飞股份有限公司 | Translation quality determining method and device |
| CN111027331B (en) * | 2019-12-05 | 2022-04-05 | 百度在线网络技术（北京）有限公司 | Method and apparatus for evaluating translation quality |
| CN111090727B (en) * | 2019-12-06 | 2022-07-08 | 思必驰科技股份有限公司 | Language conversion processing method and device and dialect voice interaction system |
-
2020
- 2020-12-31 CN CN202011637424.0A patent/CN112734205A/en active Pending
-
2021
- 2021-03-26 WO PCT/CN2021/083175 patent/WO2022141838A1/en active Application Filing
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113268895A (en) * | 2021-07-20 | 2021-08-17 | 湖南高至科技有限公司 | Combined simulation method, device, equipment and medium based on semantic mapping |
| CN113268895B (en) * | 2021-07-20 | 2021-09-21 | 湖南高至科技有限公司 | Combined simulation method, device, equipment and medium based on semantic mapping |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022141838A1 (en) | 2022-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112883190A (en) | Text classification method and device, electronic equipment and storage medium | |
| CN111814962A (en) | Method and device for acquiring parameters of recognition model, electronic equipment and storage medium | |
| CN113378970B (en) | Sentence similarity detection method and device, electronic equipment and storage medium | |
| CN113033198B (en) | Similar text pushing method and device, electronic equipment and computer storage medium | |
| CN113704429A (en) | Semi-supervised learning-based intention identification method, device, equipment and medium | |
| CN112507663A (en) | Text-based judgment question generation method and device, electronic equipment and storage medium | |
| CN113706322A (en) | Service distribution method, device, equipment and storage medium based on data analysis | |
| CN114398557A (en) | Information recommendation method and device based on double portraits, electronic equipment and storage medium | |
| CN112559687A (en) | Question identification and query method and device, electronic equipment and storage medium | |
| CN113706291A (en) | Fraud risk prediction method, device, equipment and storage medium | |
| CN113821622A (en) | Answer retrieval method and device based on artificial intelligence, electronic equipment and medium | |
| CN115238670A (en) | Information text extraction method, device, equipment and storage medium | |
| CN114880449A (en) | Reply generation method and device of intelligent question answering, electronic equipment and storage medium | |
| CN113869456A (en) | Sampling monitoring method and device, electronic equipment and storage medium | |
| CN111460293B (en) | Information pushing method and device and computer readable storage medium | |
| CN112734205A (en) | Model confidence degree analysis method and device, electronic equipment and computer storage medium | |
| CN112101481A (en) | Method, device and equipment for screening influence factors of target object and storage medium | |
| CN112269875A (en) | Text classification method and device, electronic equipment and storage medium | |
| CN114943306A (en) | Intention classification method, device, equipment and storage medium | |
| CN114741608A (en) | News recommendation method, device, equipment and storage medium based on user portrait | |
| CN113628043A (en) | Complaint validity judgment method, device, equipment and medium based on data classification | |
| CN114780688A (en) | Text quality inspection method, device and equipment based on rule matching and storage medium | |
| CN113515591A (en) | Text bad information identification method and device, electronic equipment and storage medium | |
| CN113888265A (en) | Product recommendation method, device, equipment and computer-readable storage medium | |
| CN113723114A (en) | Semantic analysis method, device and equipment based on multi-intent recognition and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40041500 Country of ref document: HK |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |