CN111125295B - LSTM-based method and system for obtaining answers to food safety questions - Google Patents
LSTM-based method and system for obtaining answers to food safety questions Download PDFInfo
- Publication number
- CN111125295B CN111125295B CN201911113762.1A CN201911113762A CN111125295B CN 111125295 B CN111125295 B CN 111125295B CN 201911113762 A CN201911113762 A CN 201911113762A CN 111125295 B CN111125295 B CN 111125295B
- Authority
- CN
- China
- Prior art keywords
- question
- questions
- answer
- food safety
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012549 training Methods 0.000 claims abstract description 44
- 239000013598 vector Substances 0.000 claims description 40
- 238000012360 testing method Methods 0.000 claims description 13
- 238000002790 cross-validation Methods 0.000 claims description 10
- 230000015654 memory Effects 0.000 claims description 9
- 238000002372 labelling Methods 0.000 claims description 6
- 230000011218 segmentation Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/316—Indexing structures
- G06F16/319—Inverted lists
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a method and a system for obtaining answers to food safety questions based on LSTM. The method comprises the following steps: acquiring a question to be answered; inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining a question matching result output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set. According to the embodiment of the invention, the food safety problem is trained based on the LSTM model, the problem that the search engine is low in retrieval accuracy of the food safety event is solved, and answers to the food safety problem can be effectively screened from a news base.
Description
Technical Field
The invention relates to the technical field of natural language processing, in particular to a method and a system for obtaining answers to food safety questions based on LSTM.
Background
In recent years, due to frequent occurrence of various food safety events, the food safety problem is more serious, and the concern of the public on the food safety is aggravated. Data for food safety class information is often difficult to obtain due to information asymmetry issues.
The corpus of food safety events at present mainly comes from news reports. However, due to the varying length of news text, combined with uncertainty in answer distribution, it is inefficient to manually find answers to a food safety related question in a food safety event news story.
While most automated question-answering systems construct a food safety database in a structured way, from which the corresponding data that matches the question is chosen as an answer to be returned to the consumer. The database construction method is very dependent on the quality of manually labeling corresponding data, and once the data is labeled with errors or answer data corresponding to questions is not labeled, reasonable information feedback can not be given to consumers. It would be desirable to construct a food-safe question-answering system that can find answers to questions from news stories, and which can produce answers by itself, rather than from an existing database.
Therefore, a method for obtaining automatic answers to questions of food safety is needed to provide a query of related food safety events and provide related sentences containing the answers, so that consumers can conveniently know the food safety events.
Disclosure of Invention
The embodiment of the invention provides a method and a system for obtaining answers to food safety questions based on LSTM (least squares), which are used for solving the problem that the answer efficiency is low by manually searching the food safety questions in the prior art.
In a first aspect, an embodiment of the present invention provides a method for obtaining answers to food safety questions based on LSTM, including:
acquiring a question to be answered;
inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining a question matching result output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set.
Preferably, the question-answer judgment model is obtained by the following steps:
acquiring a sample set of answers to the food safety questions;
classifying according to the question answer pairs, defining sentences in which the target answers are contained in the question answer pairs as first labels, and defining sentences in which the target answers are not contained in the question answer pairs as second labels;
dividing the food safety problem sample set into a training set, a cross-validation set and a test set based on the first tag and the second tag;
acquiring an LSTM model network structure as an initial model;
and inputting the training set, the cross validation set and the test set into the initial model for training to obtain the question-answer judging model.
Preferably, the obtaining a sample set of answers to the food safety questions specifically includes:
acquiring a food safety original document;
manually asking questions based on a preset food safety event, marking sentences with matching answers to the manual questions in the food safety original document, and forming matching question answer pairs by the sentences with the manual questions and the manual questions;
labeling sentences with preset non-association degrees with the manual questioning in the food safety original document, and forming a non-matching question answer pair by the sentences with the preset non-association degrees and the manual questioning;
word2Vec model is adopted to generate Word vectors in the food safety field, and the questions and answers after the Word segmentation in the question answer pair are mapped to Word vector space to generate question answer pair vector representation;
and constructing the word vector into the food safety question answer sample set.
Preferably, the manual questioning is performed based on a preset food safety event, sentences with matching answers to the manual questioning are marked in the food safety original document, and the sentences with matching answers to the manual questioning and the manual questioning form matching question answer pairs, which further comprises:
and searching an article with highest matching degree with the manual questioning in the food safety original document by using a Lucene inverted index system, and searching sentences containing information related to the manual questioning in the article.
Preferably, the inputting the questions to be answered to a pre-trained question-answer judging model, and obtaining the question matching result output by the question-answer judging model specifically includes:
searching an article with the highest matching degree with the questions to be answered in the food safety original document by adopting a Lucene inverted index system based on the questions to be answered, and taking the article as a candidate article;
all sentences in the candidate articles and the questions to be answered form candidate question answer pairs;
and inputting the candidate question answer pair vector representation to the question-answer judging model, calculating the matching degree by the question judging model, and taking the sentence with the highest matching degree as the question matching result.
Preferably, the matching degree calculation performed by the problem judgment model specifically includes:
the question judgment model performs matching degree scoring according to the matching degree of all sentences and the questions to be answered to obtain matching degree scores;
and extracting sentences from the high-order to the low-order according to the matching degree score to match the questions to be answered.
Preferably, the question answer pair is in the format of a question-answer-tag value.
In a second aspect, an embodiment of the present invention provides a system for obtaining answers to food safety questions based on LSTM, including:
the acquisition module is used for acquiring the questions to be answered;
the processing module is used for inputting the questions to be answered into a pre-trained question-answer judging model and obtaining the question matching results output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
a memory, a processor, and a computer program stored on the memory and executable on the processor, which when executed implements the steps of any of the LSTM based methods of obtaining answers to food safety questions.
In a fourth aspect, embodiments of the present invention provide a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the LSTM-based methods of obtaining answers to food safety questions.
According to the LSTM-based method and system for obtaining the answers to the food safety questions, provided by the embodiment of the invention, the LSTM-based model is used for training the food safety questions, the problem that a search engine is low in efficiency on the food safety events is solved, and the answers to the food safety questions can be effectively screened out from a news base.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for obtaining answers to food safety questions based on LSTM according to an embodiment of the present invention;
fig. 2 is a flowchart of a problem determination model training method according to an embodiment of the present invention:
FIG. 3 is a schematic diagram of an overall process of training a problem determination model according to an embodiment of the present invention;
FIG. 4 is a schematic workflow diagram of an automatic food safety questioning and answering system according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of the relationship between the average accuracy and the number of articles selected according to the embodiment of the present invention;
FIG. 6 is a system architecture diagram for obtaining answers to food safety questions based on LSTM according to an embodiment of the present invention;
fig. 7 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Fig. 1 is a flowchart of a method for obtaining answers to food safety questions based on LSTM according to an embodiment of the present invention, as shown in fig. 1, including:
s1, acquiring a question to be answered;
s2, inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining a question matching result output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set.
Specifically, the method is used for firstly preparing an answered food safety question according to a food safety hotspot focused by the user, inputting the question into a question-answer judging model which is trained in advance, outputting a question answer matched with the question to be answered, wherein the question-answer judging model is obtained by training an LSTM (Long Short-Term Memory) model according to a certain number of food safety question sample sets and classification labels.
According to the embodiment of the invention, the obtained food safety problem answer pair is trained based on the LSTM model, the problem that the search engine is low in efficiency on food safety events is solved, and the food safety problem answer can be effectively screened out from the news base.
Based on the above embodiments, fig. 2 is a flowchart of a training method for a question judgment model according to an embodiment of the present invention, and as shown in fig. 2, the question judgment model is obtained by:
101, acquiring a sample set of answers to food safety questions;
102, classifying according to the question answer pairs, defining sentences in which the question answer pairs contain target answers as first labels, and defining sentences in which the question answer pairs do not contain target answers as second labels;
103, dividing the food safety problem sample set into a training set, a cross-validation set and a test set based on the first tag and the second tag;
104, acquiring an LSTM model network structure as an initial model;
and 105, inputting the training set, the cross validation set and the test set into the initial model for training to obtain the question-answer judging model.
Specifically, in step 101, a sample set of food safety problems to be trained is obtained;
in step 102, the obtained question answer pair data is used to train an LSTM model capable of judging the matching degree of the questions and the sentences, so as to ensure the accuracy and generalization capability of the neural network, and ensure that the proportion of the matched questions and answers in the training data is 50%. The sentences of all the question answer pairs containing the answers are provided with labels=1, namely a first label, the sentences of the questions not containing the answers are provided with labels=0, namely a second label, and the trained model can give the probability of representing that the given answer candidate sentence can accurately answer the questions, namely the matching degree of the question answer pair;
in step 103, after the corresponding classification labels are divided, the sample set is divided into a training set, a cross validation set and a test set so as to facilitate the subsequent model training; in order to ensure the accuracy and generalization capability of the neural network, the proportion of whether the questions and the answers match or not in the training data is ensured to be 50 percent. Sentence token=1 for all pairs of questions answers containing the answer, sentence token=0 for no answer; all use cases are randomly arranged and divided into 7:2:1, a training set, a cross-validation set and a test set;
in step 104, an LSTM model is obtained as an initial model structure, where LSTM is a time-loop neural network, specifically designed to solve the long-term dependency problem existing in a general RNN (loop neural network), and all RNNs have a chain form of a repeating neural network module;
in step 105, the sample set is input into the obtained initial model to start training, the input layer data uniformly adopts word vectors given by the word vector model generated previously, each training case comprises question text vector representation, candidate answer sentence text vector representation and 0-1 label (indicating whether the two are matched), firstly, word vector dimension of word2vec is set to be 20, word unit number in sentences is uniform to be 80, then, for the excessive word units, truncation processing is carried out, and for the questions or answer sentences with insufficient word unit number, word vector dimension supplement is adopted to be 0, and the word unit number is expanded to 80 by using full-zero word vector filling, so that the word vector number contained in each question or answer sentence sample received by the LSTM input layer is ensured to be consistent. And then constructing a vocabulary of the training corpus, so that words in the text correspond to the pre-training vectors one by one, and designating the pre-training word vectors. In the training process, the LSTM uses an ADAM optimization method to carry out gradient descent on the training set, and the attributes such as the layer number, the learning rate and the like belong to adjustable training parameters. Training parameters are adjusted using the cross-test set, and result verification is performed using the test set. The LSTM output layer calculates the relevant probability values using a Softmax function. The trained model can calculate the matching degree of the given question and the candidate answer sentence according to the word vector representation of the given question and the candidate answer sentence, and the closer the numerical value is to 1, the greater the probability that the candidate sentence can accurately answer the question is represented, and the whole training process is schematically shown in fig. 3.
According to the embodiment of the invention, the LSTM model is introduced, and the model is trained and learned based on label classification, so that the question-answer judgment model obtained through training has strong learning ability, and the matching answers of the questions can be effectively identified and matched.
Based on any one of the above embodiments, the obtaining a sample set of answers to the food safety questions specifically includes:
acquiring a food safety original document;
manually asking questions based on a preset food safety event, marking sentences with matching answers to the manual questions in the food safety original document, and forming matching question answer pairs by the sentences with the manual questions and the manual questions;
labeling sentences with preset non-association degrees with the manual questioning in the food safety original document, and forming a non-matching question answer pair by the sentences with the preset non-association degrees and the manual questioning;
word2Vec model is adopted to generate Word vectors in the food safety field, and the questions and answers after the Word segmentation in the question answer pair are mapped to Word vector space to generate question answer pair vector representation;
and constructing the word vector into the food safety question answer sample set.
Specifically, a certain number of food safety original documents are obtained, manual questioning is carried out aiming at related food safety events, sentences where answers are located are manually marked in the documents, irrelevant document sentence contents are selected, and matched and unmatched answer pairs are respectively formed with the proposed questions. And (3) word segmentation is carried out on the texts in the answers of the questions, word2vec technology is combined, word vector models in the food safety field are used for generating word vectors, and the word vectors form a food safety question sample set.
Based on any of the above embodiments, the manually asking question based on a preset food safety event marks a sentence with an answer matching the manually asking question in the food safety original document, and forms a matching question answer pair with the manually asking question, and further includes:
and searching an article with highest matching degree with the manual questioning in the food safety original document by using a Lucene inverted index system, and searching sentences containing information related to the manual questioning in the article.
Specifically, for the obtained food safety corpus, manually proposing questions related to the content of the article, selecting related answers from the article, and combining the answer candidate sentences and the manually proposing questions into question answer pairs; the number of answer pairs of the questions can be expanded, and the specific method is as follows: and using the Lucene inverted index system to find out the article with the highest matching degree with the questions in the document corpus, and finding out all sentences containing the required answers in the documents to generate question answer pairs. And randomly selecting irrelevant sentences in the document, and forming unmatched question answer pairs with manually-proposed questions.
For example, the Lucene inverted index system is used to index all 7053 food safety articles for subsequent retrieval. The number of answer pairs of the questions can be expanded, and the specific method is as follows: using a Lucene inverted index system to find N articles with highest matching degree with the questions from the document corpus, finding all sentences containing required answers from the N articles to serve as answer candidate sentences capable of answering given questions, and generating answer pairs with the given questions; in addition, irrelevant sentences in the documents are randomly selected, and a non-matching question answer pair is formed by the irrelevant sentences and manually-proposed questions. Based on 899 question answer pairs corpus of manual labeling, 9850 question answer pairs are obtained from 7053 food safety article corpus by Lucene. Then, using a jieba word segmentation model to segment sentences in 17 ten thousand food safety news reports as basic data of a training word vector model, and mining the semantics of the vocabulary in the food safety field as far as possible so as to improve the field adaptability of the system in the food safety specific field; the word vector dimension was 20 using the Skip-Gram model in word2 vec. The window value is set to be 5, the alpha value is set to be 0.01, training is carried out by using the segmented news document, vector representation of each word is obtained, and words with similar meanings are mapped to adjacent positions in a vector space correspondingly. And for the question text and the answer candidate sentence text in all the question answer pairs, converting the segmented text into word vectors for subsequent use by using a word vector model in the food safety field based on word2vec technology.
Here, lucene is a full text search engine, which is a full text search engine toolkit of open source code, but it is not a complete full text search engine, but a full text search engine architecture, providing complete query engine and index engine, and a partial text analysis engine. The purpose of Lucene is to provide a simple and easy-to-use tool kit for software developers to conveniently realize the full-text retrieval function in a target system or to establish a complete full-text retrieval engine based on the function.
According to the embodiment of the invention, the Lucene search engine with stronger universality is adopted, the original document is initially searched, and the article with the highest matching degree is locked, so that the accuracy of problem search is effectively improved.
Based on any one of the foregoing embodiments, the inputting the to-be-answered question into a pre-trained question-answering judgment model, and obtaining a question matching result output by the question-answering judgment model specifically includes:
searching an article with the highest matching degree with the questions to be answered in the food safety original document by adopting a Lucene inverted index system based on the questions to be answered, and taking the article as a candidate article;
all sentences in the candidate articles and the questions to be answered form candidate question answer pairs;
and inputting the candidate question answer pair vector representation to the question-answer judging model, calculating the matching degree by the question judging model, and taking the sentence with the highest matching degree as the question matching result.
The matching degree calculation performed by the problem judgment model specifically includes:
the question judgment model performs matching degree scoring according to the matching degree of all sentences and the questions to be answered to obtain matching degree scores;
and extracting sentences from the high-order to the low-order according to the matching degree score to match the questions to be answered.
Specifically, for a newly presented problem of a user, all food safety documents are searched by utilizing the Lucene inverted index, and an article with the highest matching degree with the text of the problem is searched to be used as a candidate article. Traversing the articles, taking all sentences in the candidate articles as candidate sentences and forming question answer pairs with questions proposed by users, carrying out matching degree calculation by the LSTM model trained in the embodiment, and extracting sentences from high to low according to the probability that the sentences can answer the questions and taking the sentences as answers given by the LSTM model. In connection with the foregoing embodiment, the workflow of the food safety automatic question-answering system is shown in fig. 4.
The number of candidate articles is selected to affect the overall accuracy of extracting answer sentences, and in the experiment, the result is shown in fig. 5, and the highest accuracy obtained by selecting one candidate article with the highest matching degree is found.
In addition, the LSTM-based question-answering system and the traditional question-answering system based on the document retrieval technology (such as the Lucene system) are compared in terms of accuracy: at 899 question-answer pairs marked by people, 80 extensions are randomly taken out to be used as a test set, and by extracting different numbers of articles to be expanded into 33355 question-answer pairs, comparison is made on an LSTM-based question-answer system and a Lucene-based question-answer system respectively from two aspects of the number of articles selected (N) and the number of candidate sentences selected by each question (S), and the results are shown in table 1:
TABLE 1
As can be seen from the two comparison results, the average accuracy of the LSTM model matching method combined with Lucene is 7-8 percent higher than that obtained by simply using Lucene for searching (the traditional searching method based on character string matching).
According to the embodiment of the invention, by adopting the LSTM model matching method combined with Lucene, the LSTM model has relatively strong capability of finding the correct answer to the given question, so that the implementation method of the LSTM-based food safety automatic question-answering system is proved, a relatively complete automatic question-answering system can be realized, and the efficiency and accuracy of obtaining the answer to the question from the related information of massive food safety are effectively improved.
Based on any of the above embodiments, the question answer pair is in the format of a question-answer-tag value.
Here, the formats of the question answer pairs all adopt the question-answer-label values uniformly, so that the matching relation between the questions and the answers and the corresponding label values is intuitively reflected.
Fig. 6 is a system structure diagram for obtaining answers to food safety questions based on LSTM according to an embodiment of the present invention, as shown in fig. 6, including: an acquisition module 61 and a processing module 62; wherein:
the obtaining module 61 is configured to obtain a question to be answered; the processing module 62 is configured to input the question to be answered to a pre-trained question-answer judging model, and obtain a question matching result output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set.
The system provided by the embodiment of the present invention is used for executing the corresponding method, and the specific implementation manner of the system is consistent with the implementation manner of the method, and the related algorithm flow is the same as the algorithm flow of the corresponding method, which is not repeated here.
According to the embodiment of the invention, the food safety problem is trained based on the LSTM model, the problem that the search engine is low in retrieval accuracy of the food safety event is solved, and answers to the food safety problem can be effectively screened from a news base.
Fig. 7 illustrates a physical schematic diagram of an electronic device, as shown in fig. 7, which may include: processor 710, communication interface (Communications Interface) 720, memory 730, and communication bus 740, wherein processor 710, communication interface 720, memory 730 communicate with each other via communication bus 740. Processor 710 may call logic instructions in memory 730 to perform the following method: acquiring a question to be answered; inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining a question matching result output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set.
Further, the logic instructions in the memory 730 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, embodiments of the present invention further provide a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the transmission method provided in the above embodiments, for example, including: acquiring a question to be answered; inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining a question matching result output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (6)
1. A method for obtaining answers to food safety questions based on LSTM, comprising:
acquiring a question to be answered;
inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining a question matching result output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set;
the question-answer judging model is obtained through the following steps:
acquiring a sample set of answers to the food safety questions;
classifying according to the question answer pairs, defining sentences in which the target answers are contained in the question answer pairs as first labels, and defining sentences in which the target answers are not contained in the question answer pairs as second labels;
dividing the food safety problem sample set into a training set, a cross-validation set and a test set based on the first tag and the second tag;
acquiring an LSTM model network structure as an initial model;
inputting the training set, the cross validation set and the test set into the initial model for training to obtain the question-answer judging model;
the method for obtaining the answer sample set of the food safety question specifically comprises the following steps:
acquiring a food safety original document;
manually asking questions based on a preset food safety event, marking sentences with matching answers to the manual questions in the food safety original document, and forming matching question answer pairs by the sentences with the manual questions and the manual questions;
labeling sentences with preset non-association degrees with the manual questioning in the food safety original document, and forming a non-matching question answer pair by the sentences with the preset non-association degrees and the manual questioning;
word2Vec model is adopted to generate Word vectors in the food safety field, and the questions and answers after the Word segmentation in the question answer pair are mapped to Word vector space to generate question answer pair vector representation;
constructing the word vector into the answer sample set of the food safety questions;
inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining the question matching results output by the question-answer judging model, wherein the method specifically comprises the following steps of:
searching an article with the highest matching degree with the questions to be answered in the food safety original document by adopting a Lucene inverted index system based on the questions to be answered, and taking the article as a candidate article;
all sentences in the candidate articles and the questions to be answered form candidate question answer pairs;
inputting the candidate question answer pair vector representation to the question-answer judging model, calculating the matching degree by the question judging model, and taking a sentence with the highest matching degree as the question matching result;
the matching degree calculation performed by the problem judgment model specifically includes:
the question judgment model performs matching degree scoring according to the matching degree of all sentences and the questions to be answered to obtain matching degree scores;
and extracting sentences from the high-order to the low-order according to the matching degree score to match the questions to be answered.
2. The LSTM based method of claim 1, wherein the manually asking questions based on the preset food safety event marks sentences with matching answers to the manually asking questions in the food safety original document, and the manually asking sentences with matching answers to the manually asking questions and the manually asking questions form matching question answer pairs, further comprising:
and searching an article with highest matching degree with the manual questioning in the food safety original document by using a Lucene inverted index system, and searching sentences containing information related to the manual questioning in the article.
3. The method of claim 1 or 2, wherein the question-answer pair is in the form of a question-answer-label value.
4. A LSTM based system for obtaining answers to food safety questions comprising:
the acquisition module is used for acquiring the questions to be answered;
the processing module is used for inputting the questions to be answered into a pre-trained question-answer judging model and obtaining the question matching results output by the question-answer judging model; the question-answer judging model is obtained by training an LSTM model by using manually marked question answer pairs and corresponding matching labels based on a food safety question answer sample set;
the question-answer judging model is obtained through the following steps:
acquiring a sample set of answers to the food safety questions;
classifying according to the question answer pairs, defining sentences in which the target answers are contained in the question answer pairs as first labels, and defining sentences in which the target answers are not contained in the question answer pairs as second labels;
dividing the food safety problem sample set into a training set, a cross-validation set and a test set based on the first tag and the second tag;
acquiring an LSTM model network structure as an initial model;
inputting the training set, the cross validation set and the test set into the initial model for training to obtain the question-answer judging model;
the method for obtaining the answer sample set of the food safety question specifically comprises the following steps:
acquiring a food safety original document;
manually asking questions based on a preset food safety event, marking sentences with matching answers to the manual questions in the food safety original document, and forming matching question answer pairs by the sentences with the manual questions and the manual questions;
labeling sentences with preset non-association degrees with the manual questioning in the food safety original document, and forming a non-matching question answer pair by the sentences with the preset non-association degrees and the manual questioning;
word2Vec model is adopted to generate Word vectors in the food safety field, and the questions and answers after the Word segmentation in the question answer pair are mapped to Word vector space to generate question answer pair vector representation;
constructing the word vector into the answer sample set of the food safety questions;
inputting the questions to be answered into a pre-trained question-answer judging model, and obtaining the question matching results output by the question-answer judging model, wherein the method specifically comprises the following steps of:
searching an article with the highest matching degree with the questions to be answered in the food safety original document by adopting a Lucene inverted index system based on the questions to be answered, and taking the article as a candidate article;
all sentences in the candidate articles and the questions to be answered form candidate question answer pairs;
inputting the candidate question answer pair vector representation to the question-answer judging model, calculating the matching degree by the question judging model, and taking a sentence with the highest matching degree as the question matching result;
the matching degree calculation performed by the problem judgment model specifically includes:
the question judgment model performs matching degree scoring according to the matching degree of all sentences and the questions to be answered to obtain matching degree scores;
and extracting sentences from the high-order to the low-order according to the matching degree score to match the questions to be answered.
5. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor performs the steps of the LSTM based method of obtaining answers to food safety questions as claimed in any one of claims 1 to 3.
6. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of the LSTM based method of obtaining answers to food safety questions as claimed in any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911113762.1A CN111125295B (en) | 2019-11-14 | 2019-11-14 | LSTM-based method and system for obtaining answers to food safety questions |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911113762.1A CN111125295B (en) | 2019-11-14 | 2019-11-14 | LSTM-based method and system for obtaining answers to food safety questions |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111125295A CN111125295A (en) | 2020-05-08 |
| CN111125295B true CN111125295B (en) | 2023-11-24 |
Family
ID=70495629
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911113762.1A Active CN111125295B (en) | 2019-11-14 | 2019-11-14 | LSTM-based method and system for obtaining answers to food safety questions |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111125295B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112131353A (en) * | 2020-10-12 | 2020-12-25 | 吉林大学 | Method for solving full text interpretation analysis problem |
| CN112580343A (en) * | 2020-11-03 | 2021-03-30 | 北京字节跳动网络技术有限公司 | Model generation method, question and answer quality judgment method, device, equipment and medium |
| CN113159187A (en) * | 2021-04-23 | 2021-07-23 | 北京金山数字娱乐科技有限公司 | Classification model training method and device, and target text determining method and device |
| CN113569025A (en) * | 2021-07-23 | 2021-10-29 | 上海明略人工智能(集团)有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108345585A (en) * | 2018-01-11 | 2018-07-31 | 浙江大学 | A kind of automatic question-answering method based on deep learning |
| CN109766423A (en) * | 2018-12-29 | 2019-05-17 | 上海智臻智能网络科技股份有限公司 | Answering method and device neural network based, storage medium, terminal |
| CN110222152A (en) * | 2019-05-29 | 2019-09-10 | 北京邮电大学 | The problem of one kind is understood based on machine reading answer acquisition methods and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107220380A (en) * | 2017-06-27 | 2017-09-29 | 北京百度网讯科技有限公司 | Question and answer based on artificial intelligence recommend method, device and computer equipment |
-
2019
- 2019-11-14 CN CN201911113762.1A patent/CN111125295B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108345585A (en) * | 2018-01-11 | 2018-07-31 | 浙江大学 | A kind of automatic question-answering method based on deep learning |
| CN109766423A (en) * | 2018-12-29 | 2019-05-17 | 上海智臻智能网络科技股份有限公司 | Answering method and device neural network based, storage medium, terminal |
| CN110222152A (en) * | 2019-05-29 | 2019-09-10 | 北京邮电大学 | The problem of one kind is understood based on machine reading answer acquisition methods and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111125295A (en) | 2020-05-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111125295B (en) | LSTM-based method and system for obtaining answers to food safety questions | |
| CN110502621B (en) | Question answering method, question answering device, computer equipment and storage medium | |
| CN111475623B (en) | Case Information Semantic Retrieval Method and Device Based on Knowledge Graph | |
| CN117033608B (en) | Knowledge graph generation type question-answering method and system based on large language model | |
| CN112667794A (en) | Intelligent question-answer matching method and system based on twin network BERT model | |
| CN102663129A (en) | Medical field deep question and answer method and medical retrieval system | |
| CN112650840A (en) | Intelligent medical question-answering processing method and system based on knowledge graph reasoning | |
| US20140351228A1 (en) | Dialog system, redundant message removal method and redundant message removal program | |
| CN111563384B (en) | Evaluation object identification method and device for E-commerce products and storage medium | |
| CN112328800A (en) | System and method for automatically generating programming specification question answers | |
| CN111078837A (en) | Intelligent question and answer information processing method, electronic equipment and computer readable storage medium | |
| CN107247751B (en) | LDA topic model-based content recommendation method | |
| CN109522397B (en) | Information processing method and device | |
| CN113887930B (en) | Question-answering robot health evaluation method, device, equipment and storage medium | |
| CN116561538A (en) | Question-answer scoring method, question-answer scoring device, electronic equipment and storage medium | |
| CN117077792B (en) | Knowledge graph-based method and device for generating prompt data | |
| CN110969005A (en) | Method and device for determining similarity between entity corpora | |
| CN107783958B (en) | Target statement identification method and device | |
| CN117216221A (en) | Intelligent question-answering system based on knowledge graph and construction method | |
| CN111782789A (en) | Intelligent question and answer method and system | |
| CN116362331A (en) | Knowledge point filling method based on man-machine cooperation construction knowledge graph | |
| CN108241650B (en) | Training method and device for training classification standard | |
| CN114443846A (en) | Classification method and device based on multi-level text abnormal composition and electronic equipment | |
| CN115374259A (en) | Question and answer data mining method and device and electronic equipment | |
| CN113468311A (en) | Knowledge graph-based complex question and answer method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |