CN117633643A - Automatic middle school geometric problem solving method based on contrast learning - Google Patents

Abstract

The invention discloses an automatic solving method of a middle school geometric problem based on contrast learning, which comprises the following steps: collecting a plurality of middle school geometric questions and answers to obtain a required middle school geometric data set; dividing the geometric problem data into a non-image geometric problem data set and an image geometric problem data set; inputting the non-graph geometric questions into a geometric image generator to obtain a geometric figure with high accuracy; inputting the multi-modal feature vector into a graphic thematic device to obtain a final multi-modal feature vector; a program decoder in the graphic thematic device obtains a solution question answer with high accuracy; and finally, the geometric image generator and the graphic problem solving device are tested together to form a unified large model for solving the self-drawing and graphic geometric problem types. The beneficial effects of the invention are as follows: from a brand new view point, the geometric problem of middle school is divided into two types of questions to be respectively and correspondingly solved, and the two types of questions are fused together to form a model capable of solving the geometric problem of middle school.

Description

An automatic solution method for middle school geometry problems based on comparative learning

Technical field

The invention relates to the field of machine learning systems, and is specifically an automatic solution method for middle school geometry problems based on comparative learning.

Background technique

In recent years, the machine learning system developed by researchers to automatically solve mathematical word problems (MWP) has attracted increasing attention due to its high academic value and huge application potential in smart education. Most existing methods focus on solving arithmetic and algebraic problems, including traditional machine learning methods and network-based models, while solving geometric problems has been rarely studied. As a classic mathematical problem, geometry occupies a large part in secondary education. Due to the challenging nature of geometric problems and data characteristics, it can also serve as a multimodal numerical reasoning benchmark that requires joint reasoning on diagrams and text.

Generally speaking, a typical geometry question mainly consists of text and geometric figures. Compared with mathematics word problems that only involve problem text, geometry problem types pose the following new challenges. First, the attached problem graph provides basic information missing from the problem text, such as the relative positions of lines and points; therefore, the solver should have the ability to parse relational graphs. Second, to solve a geometric problem, we need to understand and align the semantics of text and diagrams simultaneously. However, question texts often contain some ambiguous references and implicit relationships to primitives, which increases the difficulty of joint reasoning between text and primitives. Third, many geometric problems require additional theorem knowledge during the solution process. Although some methods have tried to solve the above problems in the past, the performance of their geometric problem solving systems is far from satisfactory. They are highly dependent on limited handcrafted rules and are only validated on small-scale datasets, which makes it difficult to generalize to more complex and real-world situations. Furthermore, the complexity of the solution process means that it is difficult for humans to understand and verify its reliability.

Recently, many works have proposed unified models for various visual language reasoning and generation tasks, since the underlying visual/language understanding and reasoning capabilities are largely common. Inspired by mainstream advances, we believe that a unified geometric problem solving model is also necessary. First of all, geometry questions are divided into question types that have their own geometric figures, commonly known as geometry questions with figures; and question types that do not have graphics in the original text and require drawings to assist in solving the problem, commonly known as geometry questions without figures. Regardless of the type of geometry questions, some basic skills and knowledge in geometric reasoning are shared in the problem-solving process. Therefore, exploring the general understanding and reasoning capabilities of unified neural networks in the field of mathematics is a very meaningful topic. Furthermore, the unified model does not require an auxiliary model to determine whether a problem is a graph geometry problem or a graph geometry problem. This greatly improves the efficiency of problem solving and reduces errors caused by classifying problem types, allowing the model to better complete problem-solving tasks. For this reason, it is valuable and worth looking forward to establishing a framework that handles geometric problems uniformly at the data layer and model layer.

Contents of the invention

In order to solve the above technical problems, the present invention proposes an automatic solution method for middle school geometry problems based on comparative learning. From the perspective of question types, the geometry question types are divided into two types: geometry questions with figures and geometry questions without figures. In view of the above Corresponding solutions were proposed for the two types of questions, and a unified model of middle school geometry and mathematics that could solve these two types of questions was finally formed.

The technical solution adopted by the present invention is as follows: an automatic solution method for middle school geometry problems based on comparative learning, and the method steps are as follows:

Step S1, data set construction: collect several middle school geometry questions and answers; and divide the collected geometry questions and answers according to the training set, verification set, and test set to obtain the required middle school geometry data set;

Step S2, formal definition of the task: given a middle school geometry data set containing N pieces, it is divided into a middle school geometry problem data set without pictures and a middle school geometry problem data set C with pictures through a question type classifier;

Step S3, there are y geometry question types b y in the middle school geometry question data set B without figures that require drawings by oneself, and _z geometry question types c z in the middle school geometry question data set C with figures that have their own graphics, _and are input to middle school geometry. In the BERT feature encoder in the automatic problem solving model; obtain all word embedding feature vectors in the middle school geometry question stem;

Step S4: Input the word embedding feature vector in the stem of the middle school geometry question without pictures obtained by the BERT feature encoder into the geometric image generator. The geometric image generator is trained and fine-tuned based on the contrastive learning model and used to generate middle school geometry questions. For the required geometric figures, manual supervision is used for comparative learning model training. The geometric figure generation loss L _prior is calculated through the mean square error loss function, and the parameters of the BERT feature encoder and geometric image generator are optimized and updated to obtain the geometric figure;

Step S5: Input the word embedding feature vectors in the stems of the geometry questions in the middle school with pictures obtained by the BERT feature encoder and the corresponding geometric figures generated by the geometry questions in the middle school without pictures and the geometric image generator into the problem solver with pictures, and then The graphics encoder included in the graphic problem solver encodes the geometric figures and extracts features, and performs the alignment operation between the middle school geometry question stem and the geometric figures with the word embedding feature vector of the BERT feature encoder to obtain the final multi-modal feature vector. ;

Step S6, the program decoder in the graphical problem solver sequentially generates problem-solving programs under the guidance of multi-modal feature vectors, and uses the negative log-likelihood loss function to calculate the generation loss L _g of problem-solving errors, obtaining a high accuracy The solution to the problem;

Step S7: Combine the geometric image generator and the problem solver with pictures to test together to form a unified large model that can solve not only geometry problems without pictures that require drawing by yourself, but also geometry problems with own graphics.

Further, in step S1, the data set is constructed, a number of middle school geometry questions and answers are collected, and the following tasks are performed; specifically:

Step S11, remove duplicate middle school geometry question types and answers;

Step S12, classify middle school geometry questions and answers into two types of geometry questions. Those with figures are automatically classified as geometry questions with pictures, and those with only question stems but no geometric figures are classified as geometry questions without pictures;

Step S13, perform classification inspection on middle school geometry question types, that is, manual inspection and verification are performed on the classification results of the same middle school geometry question type;

Step S14, after manual inspection and verification, the middle school geometry questions and answers are divided according to the ratio of training set: validation set: test set = 8:1:1;

Step S15: After the middle school geometry question types and answers are divided, the middle school geometry question types of the training set and the verification set are manually annotated; the problem-solving steps are extracted based on the answers, and the problem-solving steps are manually annotated into computer recognized programming language.

Further, the BERT feature encoder in step S3 uses the encoder module in the Transformer model architecture and is composed of multi-layer bidirectional encoders. The calculation process is as shown in formula (1);

(1);

Among them, e _i ^w is the corresponding word embedding feature vector obtained by the BERT feature encoder of the i-th word token w _i .

Further, the geometric image generator in step S4 includes:

Step S41, input data into the geometric image generator, where the input data is the corresponding word embedding feature vector in the geometry question stem of the middle school without pictures;

Step S42, the geometric image generator is an adapted contrastive learning model. The contrastive learning model is a text-image pair classification model. The contrastive learning model is trained and fine-tuned to achieve the downstream task of generating geometric images;

Step S43, train the contrastive learning model:

Collect and organize the data set of geometry questions with figures to obtain middle school geometry questions and corresponding geometric figures;

Feature extraction was performed on the middle school geometry question stem and the corresponding geometric figures respectively to obtain the word embedding feature vector e _i ^w and the geometric figure feature h ^CNN to form a text-image pair;

Input the characteristics of text-image pairs into the classification model of text-image pairs for comparative learning. Under manual supervision, the text-image pairs that match each other are marked as positive samples, and the unmatched text-image pairs are marked as positive samples. is a negative sample;

The classification model of text-image pairs can obtain middle school geometry questions and corresponding geometric figures through positive samples and negative samples, that is, a word embedding feature vector e _i ^w is given to find the corresponding geometric figure feature h ^CNN ;

Step S44, fine-tune the contrastive learning model:

Define the text as x, the geometric figure as y, and introduce the resulting graphics code into Prior: , the calculation process is shown in formula (2), in which the graphic code generated by Prior is obtained by training the graphic code of the contrastive learning model as a true value;

(2);

Among them, P(y|x) represents the generation of geometric figure y based on x in this article; It is expressed that after training by the contrastive learning model, the geometric feature h ^CNN can be generated according to the word embedding feature vector e _i ^w ; Indicates that the geometric feature h ^CNN is found based on the text x, and then the geometric feature h ^CNN is decoded to generate the geometric figure y; It is expressed as finding the geometric figure feature h ^CNN based on the text x and the corresponding geometric figure y after decoding, It is expressed as finding the geometric feature h ^CNN based on the text x;

Step S45, geometric figure generation loss L _prior : According to the Prior in the fine-tuning of the contrastive learning model, the mean square error loss function is used to predict the geometric figures. The calculation process is as shown in formula (3);

(3);

Among them, L _prior represents the geometry generation loss, Indicates the summation of the previous i geometric figure generation losses, T is the number of times, h _(i) ^CNN is expressed as the i-th generated geometric figure feature h ^CNN ; Represents the geometric feature h _(i) ^CNN generated for the i-th time using text x; Indicates the difference between the geometric feature h _(i) generated by text x for the i-th time between ^CNN and geometric feature h ^CNN , where is an adjustable parameter amount.

Furthermore, step S5 includes a graphical problem solver, which includes four modules: bidirectional LSTM layer, graph encoder, joint reasoning module and program decoder; specific contents include:

Step S51, input data into the problem solver with pictures. The input data includes the word embedding feature vectors in the stems of the geometry questions in the middle school with pictures obtained by the BERT feature encoder and the corresponding character embedding feature vectors in the geometry questions in the middle school without pictures and generated by the geometric image generator. geometric figures;

Step S52, bidirectional LSTM layer: The word embedding feature vector e _i ^w in the middle school geometry question stem with pictures obtained by the BERT feature encoder is input into the bidirectional LSTM layer, and the bidirectional LSTM layer is used to obtain the corresponding character of the i-th word in the mathematical geometry text. The contextual semantic feature vector, that is, the word embedding feature vector e _i ^w , is input into the forward LSTM layer and the backward LSTM layer respectively, as shown in formula (4);

(4);

Among them, h _i ^LSTM is the contextual semantic feature vector corresponding to the i-th word in the mathematical geometry text, LSTM _f and LSTM _b respectively represent the output vector of the forward LSTM layer and the output vector of the backward LSTM layer. Represents cascading operations;

Step S53, graphics encoder: Use CNN convolutional neural network to extract geometric image features h ^CNN . The CNN convolutional neural network includes convolutional layers, nonlinear activation functions and pooling layer components; in the convolutional layer by sliding The convolution kernel performs a convolution operation on the geometric image to capture local features; at the same time, a nonlinear activation function is introduced to increase the expressive ability of the CNN convolutional neural network; the pooling layer component reduces the dimension of the feature map and retains the key features of the geometric figure; multi-layer The stacked convolutional layers enable the CNN convolutional neural network to gradually extract higher-level geometric features; the geometric feature h ^CNN is obtained through the fully connected layer;

Step S54, joint reasoning module: fuse the contextual semantic feature vector h _i ^LSTM corresponding to the i-th word with the geometric feature h ^CNN through the attention mechanism to achieve cross-border semantic fusion and alignment, and obtain the i-th word containing the attention mechanism The corresponding contextual semantic feature vector h _i ^LSTM and the multi-modal feature vector M _i corresponding to the i-th word of the geometric feature h ^CNN information, the calculation process is as follows: Formula (5) and Formula (6);

(5);

(6);

Among them, Attention represents the attention mechanism, Q, K, V represent the query vector, key vector and value vector respectively, Softmax is the normalized exponential function, dd is the second dimension size of the query vector Q and the key vector K, , , Respectively represent the projection parameter matrix of the query vector Q, key vector K and value vector V corresponding to the i-th word in the self-attention mechanism; let , ,in The parameter matrix learned for the linear layer, D represents the transpose;

Step S55, program decoder: the multi-modal feature vector Mi _is fed into the linear layer to obtain the initial state s ₀ , the hidden state s _t of the bidirectional LSTM layer at time step t is cascaded with the attention result, and is fed to the linear layer with the Softmax function Linear layer to predict the distribution of the next program token P _t .

Further, the generation loss L _g in step S6 adopts the negative log likelihood of the target program, and its calculation formula is as shown in (7);

)(7);

Among them, θ is the parameter of the loss function, P _t is the program token, y _t is the target program to be generated at time t, y _t-1 is the target program to be generated at time t-1, M _i is the multi-modal feature vector .

Furthermore, the automatic solution model for middle school geometry problems of the present invention is divided into four modules: BERT feature encoder, geometric image generator, graphic problem solver and unified large model. The BERT feature encoder serially connects the geometric image generator and The graphical problem solver, the geometric image generator and the graphical problem solver have a parallel structure, and then the large model is unified in series.

The beneficial effects of the present invention are: (1) First, collect data sets from junior high school mathematics textbooks and test papers of the People's Education Press, clean and normalize the questions and answers in the data set, thereby constructing a middle school geometry data set. Then, the middle school geometry data set is classified into two question types through the program. After the classification is completed, manual supervision and inspection are performed to verify the rationality of the classification of the present invention. Secondly, process the two classified question types separately. For graphed geometry questions that have their own graphics in the question: use a graphed problem solver model to solve them; for non-pictured geometry questions that have no graphics in the question and require your own drawing: this type of question requires the help of comparison Learning model, and training and fine-tuning the comparative learning model so that it can better generate the geometric figures required for the text in the question. Then, the geometric figures generated by the geometric image generator and the questions from the geometry questions without figures are input into the figure solver for model testing. Finally, the two trained modules are merged together to form a unified large model that can solve the above two geometric problem types at the same time.

(2) For geometric problems without drawings that need to be solved by drawing, the present invention adopts the method of fine-tuning and pre-training the comparative learning model to generate the corresponding geometric figures in the problems, realizing the transformation from "without pictures" to "with pictures" This process paved the way for subsequent problem solving.

(3) For graph geometry problems solved with built-in graphics, the present invention first uses encoders to extract features from text and geometric figures respectively, and then uses a collaborative attention mechanism to perform cross-border semantic fusion and alignment, and finally solves the cross-modal problem. The problem of joint reasoning.

(4) From a new perspective, this invention divides middle school geometry problems into two types of questions to solve respectively, and integrates them to form a unified large model that can solve middle school geometry problems.

Description of drawings

Figure 1 is a flow chart of the overall model structure of the present invention.

Detailed ways

The present invention works and is implemented in this way. It is an automatic solution method for middle school geometry problems based on comparative learning. The method steps are as follows:

Step S1, data set construction: collect several middle school geometry questions and answers; and divide the collected geometry questions and answers according to the training set, verification set, and test set to obtain the required middle school geometry data set;

Step S2, formal definition of the task: given a middle school geometry data set containing N pieces, it is divided into a middle school geometry problem data set without pictures and a middle school geometry problem data set C with pictures through a question type classifier;

Step S3, there are y geometry question types b y in the middle school geometry question data set B without figures that require drawings by oneself, and _z geometry question types c z in the middle school geometry question data set C with figures that have their own graphics, _and are input to middle school geometry. In the BERT feature encoder in the automatic problem solving model; obtain all word embedding feature vectors in the middle school geometry question stem;

Step S4: Input the word embedding feature vector in the stem of the middle school geometry question without pictures obtained by the BERT feature encoder into the geometric image generator. The geometric image generator is trained and fine-tuned based on the contrastive learning model and used to generate middle school geometry questions. For the required geometric figures, manual supervision is used for comparative learning model training. The geometric figure generation loss L _prior is calculated through the mean square error loss function, and the parameters of the BERT feature encoder and geometric image generator are optimized and updated to obtain the geometric figure;

Step S5: Input the word embedding feature vectors in the stems of the geometry questions in the middle school with pictures obtained by the BERT feature encoder and the corresponding geometric figures generated by the geometry questions in the middle school without pictures and the geometric image generator into the problem solver with pictures, and then The graphics encoder included in the graphic problem solver encodes the geometric figures and extracts features, and performs the alignment operation between the middle school geometry question stem and the geometric figures with the word embedding feature vector of the BERT feature encoder to obtain the final multi-modal feature vector. ;

Step S6, the program decoder in the graphical problem solver sequentially generates problem-solving programs under the guidance of multi-modal feature vectors, and uses the negative log-likelihood loss function to calculate the generation loss L _g of problem-solving errors, obtaining a high accuracy The solution to the problem;

Step S7: Combine the geometric image generator and the problem solver with pictures to test together to form a unified large model that can solve not only geometry problems without pictures that require drawing by yourself, but also geometry problems with own graphics.

Further, in step S1, the data set is constructed, and 16,201 geometry questions and answers are manually collected. The geometry questions and answers come from the New People's Education Press middle school textbooks, test paper syllabus and teaching plan materials, and the following tasks are performed; specifically:

Step S11, remove duplicate geometry question types and answers;

Step S12, classify the geometry question types and answers into two geometry question types. Those with figures are automatically classified as geometry questions with figures, and those with only question stems but no geometric figures are classified as geometry questions without figures;

Step S13, perform classification inspection on the geometry question type, that is, use manual inspection to verify the classification results of the same geometry question type to ensure that the classification is reasonable;

Step S14, after manual inspection and verification, 14,334 geometry questions and answers are retained. There are 9,922 geometry questions with figures automatically classified as geometry questions with figures, and 4,412 geometry questions without geometric figures automatically classified as geometry questions without figures. According to The ratio of training set: validation set: test set = 8:1:1 divides the geometry questions and answers;

Step S15: After dividing the geometric question types and answers, manually label the geometric question types of the training set and the verification set; extract the problem-solving steps based on the answers, and then manually label the problem-solving steps. into a programming language that computers can recognize.

Further, the BERT feature encoder in step S3 uses the encoder module in the Transformer model architecture and is composed of multi-layer bidirectional encoders. The calculation process is as shown in formula (1);

(1);

Among them, e _i ^w is the corresponding word embedding feature vector obtained by the BERT feature encoder of the i-th word token w _i .

Further, the geometric image generator in step S4 includes:

Step S41, input data into the geometric image generator, where the input data is the corresponding word embedding feature vector in the question stem of the non-picture geometry question;

Step S42, the geometric image generator is an adapted contrastive learning model. The contrastive learning model is a text-image pair classification model. The contrastive learning model is trained and fine-tuned to achieve the downstream task of generating geometric images;

Step S43, train the contrastive learning model:

Collected and organized a data set of geometry questions with figures, and obtained 9922 geometry questions and corresponding geometric figures;

Feature extraction was performed on 9922 geometric question stems and corresponding geometric figures to obtain word embedding feature vectors e _i ^w and geometric figure features h ^CNN , forming 9922 text-image pairs;

Input the features of 9922 text-image pairs into the classification model of text-image pairs for comparative learning. Under manual supervision, the text-image pairs that match each other are marked as positive samples, and the unmatched text-image pairs are marked as positive samples. Mark as negative sample;

The classification model of text-image pairs can obtain the geometric question stem and the corresponding geometric figure through positive samples and negative samples, that is, a word embedding feature vector e _i ^w is given to find the corresponding geometric figure feature h ^CNN ;

Step S44, fine-tune the contrastive learning model:

Define the text as x, the geometric figure as y, and introduce the resulting graphics code into Prior: , the calculation process is shown in formula (2), in which the graphic code generated by Prior is obtained by training the graphic code of the contrastive learning model as a true value;

(2);

Among them, P(y|x) represents the generation of geometric figure y based on this article x; It is expressed that after training by the contrastive learning model, the geometric feature h ^CNN can be generated according to the word embedding feature vector e _i ^w ; Indicates that the geometric feature h ^CNN is found based on the text x, and then the geometric feature h ^CNN is decoded to generate the geometric figure y; It is expressed as finding the geometric figure feature h ^CNN based on the text x and its decoded corresponding geometric figure y, It is expressed as finding the geometric feature h ^CNN based on the text x;

Step S45, geometric figure generation loss L _prior : According to the Prior in the fine-tuning of the contrastive learning model, the mean square error loss function is used to predict the geometric figures. The calculation process is as shown in formula (3);

(3);

Among them, L _prior represents the geometry generation loss, Indicates the summation of the previous i geometric figure generation losses, T is the number of times, h _(i) ^CNN is expressed as the i-th generated geometric figure feature; Represents the geometric feature h _(i) ^CNN generated for the i-th time using text x; Indicates the difference between the geometric feature h _(i) generated by text x for the i-th time between ^CNN and geometric feature h ^CNN , where is an adjustable parameter amount.

Furthermore, step S5 includes a graphical problem solver, which includes four modules: bidirectional LSTM layer, graph encoder, joint reasoning module and program decoder; specific contents include:

Step S51, input data into the problem solver with pictures. The input data includes the word embedding feature vectors in the stems of the geometry questions with pictures obtained by the BERT feature encoder and the corresponding geometric figures generated by the geometry questions without pictures and the geometric image generator. ;

Step S52, Bidirectional LSTM layer: The word embedding feature vector e _i ^w in the question stem of the graph geometry question obtained by the BERT feature encoder is input into the bidirectional LSTM layer, and the bidirectional LSTM layer is used to obtain the context corresponding to the i-th word in the mathematical geometry text. The semantic feature vector, that is, the word embedding feature vector e _i ^w , is input into the forward LSTM layer and the backward LSTM layer respectively, as shown in formula (4);

(4);

Among them, h _i ^LSTM is the contextual semantic feature vector corresponding to the i-th word in the mathematical geometry text, LSTM _f and LSTM _b respectively represent the output vector of the forward LSTM layer and the output vector of the backward LSTM layer. Represents cascading operations;

Step S53, graphics encoder: Use CNN convolutional neural network to extract image features h ^CNN . Specifically, the feature extraction of geometric figures is achieved through components such as convolution layers, nonlinear activation functions, and pooling layers. In the convolution layer, the geometric image is convolved by sliding convolution kernels to capture local features such as edge lines and points. At the same time, a nonlinear activation function is introduced to increase the expressive ability of the network. The pooling layer reduces the dimensionality of the feature map and retains the key features of the geometry. Multiple stacked convolutional layers enable the network to gradually extract higher-level geometric features, such as the shape of geometric figures and the positional relationships of points and lines. Finally, the features of the geometric figures are obtained through the fully connected layer;

Step S54, joint reasoning module: use the attention mechanism to fuse the contextual semantic feature vector h _i ^LSTM corresponding to the i-th word with the geometric feature h ^CNN to achieve cross-border semantic fusion and alignment, and obtain the i-th word containing the attention mechanism Corresponding contextual semantic feature vector h _(i) ^CNN and geometric feature h Multi-modal feature vector M _i corresponding to the i-th word of ^CNN information, the calculation process is as follows: Formula (5) and Formula (6);

(5);

(6);

Among them, Attention represents the attention mechanism, Q, K, V represent the query vector, key vector and value vector respectively, Softmax is the normalized exponential function, dd is the second dimension size of the query vector Q and the key vector K, , , Respectively represent the projection parameter matrix of the query vector Q, key vector K and value vector V corresponding to the i-th word in the self-attention mechanism; let , ,in The parameter matrix learned for the linear layer, D represents the transpose;

Step S55, program decoder: the multi-modal feature vector Mi _is fed into the linear layer to obtain the initial state s ₀ , the hidden state s _t of the bidirectional LSTM layer at time step t is cascaded with the attention result, and is fed to the linear layer with the Softmax function Linear layer to predict the distribution of the next program token P _t .

Further, the generation loss L _g in step S6 adopts the negative log likelihood of the target program, and its calculation formula is as shown in (7);

)(7);

Among them, θ is the parameter of the loss function, P _t is the program token, y _t is the target program to be generated at time t, y _t-1 is the target program to be generated at time t-1, M _i is the multi-modal feature vector .

Furthermore, the automatic solution model for middle school geometry problems of the present invention is divided into four modules: BERT feature encoder, geometric image generator, graphic problem solver and unified large model. The BERT feature encoder serially connects the geometric image generator and The graphical problem solver, the geometric image generator and the graphical problem solver have a parallel structure, and then the large model is unified in series.

Furthermore, the unified large model is trained by the geometric image generator and the problem solver with pictures, and then the large model is tested. If a middle school geometry question is input, regardless of whether it has pictures or no pictures, if it is a geometry question with pictures, Then it is directly passed through the feature encoder and then input into the problem solver with a graph for problem solving; and if it is a geometry problem without a graph, the geometric image generator can generate a geometric image to turn it into a geometry problem with a graph and then input it into the graph problem solver. Solve it in the problem solver. So in summary, we have created a unified large model that can handle both graph geometry problems with built-in graphics and graph-free geometry problems without geometric shapes.

As shown in Figure 1, Figure 1 is a flow chart of model prediction. The process is as follows: input the N-channel middle school geometry data set into the automatic solving model for middle school geometry problems, and then classify the N-channel middle school geometry into geometry problems without figures and geometry problems with figures. Geometry questions; for graph-free geometry questions, the features are first extracted by the BERT feature encoder and then input into the geometric image generator. The geometric image generator can generate questions according to the meaning of the question through comparative learning, training and fine-tuning. The geometric figure described in , where the prediction of the figure uses the mean square error loss function to obtain the prediction loss L _prior , thereby optimizing and updating the parameters in the geometric image generator, improving the accuracy of the generated graphics, and finally generating the geometric image generator The geometric figures and non-graph geometry problems are input into the problem solver with graphs for solution testing; for the geometry problems with graphs, they are first extracted by the BERT feature encoder, and then continue to be input into the problem solver with graphs for solving, where During the training period, the generation loss L _g is the negative log likelihood of the target program, which can improve the accuracy of the graphical problem solver; during the training stage, the middle school geometry problem automatic solving model will calculate the joint total loss L = L _prior + L _g , to simultaneously optimize the parameters in the problem solver without graphics and the problem solver with graphics, and enhance the information interaction between the two modules; in the testing phase, the geometric image generator and the problem solver with graphics are integrated to form a solution A unified large model for these two types of geometric questions.

Claims (7)

1. An automatic solving method of a middle school geometric problem based on contrast learning is characterized by comprising the following steps: the method comprises the following steps:

step S1, constructing a data set: collecting a plurality of geometric questions and answers of the middle school; dividing the collected multiple geometric questions and answers according to the training set, the verification set and the test set respectively to obtain a required geometric data set of middle school;

step S2, task formalization definition: given a medium geometry dataset comprising N pieces, it is divided into: a non-pictorial geometric topic data set B and a pictorial geometric topic data set C;

step S3, there are y geometric questions B needing to be drawn by oneself in the study geometric question data set B in the no-graph _y And geometric topic C with z-channel self-graphic in geometric topic data set C in graph _z Input to automatic solving of geometric problems in middle schoolIn the BERT signature encoder in the solution model; acquiring embedded feature vectors of all words in the middle school geometric stem;

s4, inputting word embedded feature vectors in the non-graphic middle school geometric question stems obtained by the BERT feature encoder into a geometric image generator, wherein the geometric image generator is obtained based on comparison learning model training and fine tuning and is used for generating geometric figures required by the middle school geometric questions, the comparison learning model training is carried out by adopting a manual supervision mode, and geometric figure generation loss L is calculated by means of a mean square error loss function _prior Optimizing and updating parameters of the BERT feature encoder and the geometric image generator to obtain geometric figures;

s5, inputting word embedded feature vectors in the geometric problem stems with graphics obtained by the BERT feature encoder and geometric figures generated by the geometric problem and geometric image generator without graphics into the graphic problem device, then encoding the geometric figures by the graphic encoder contained in the graphic problem device, extracting features, and performing alignment operation of the geometric problem stems with the geometric figures with the word embedded feature vectors of the BERT feature encoder to obtain final multi-mode feature vectors;

step S6, the program decoder in the diagrammatical question device sequentially generates a question solving program under the guidance of the multi-mode feature vector, and calculates the generation loss L of the question solving error by adopting the negative log likelihood loss function _g Obtaining a question-solving answer with high accuracy;

and S7, testing the geometric image generator and the graphic problem solving device together to form a unified large model which can solve the problem of the graphic-free geometric problem needing to be drawn by the user and the geometric problem with the graphic.

2. The automatic middle school geometric problem solving method based on contrast learning according to claim 1, wherein the method is characterized in that: step S1, constructing a data set, collecting geometric questions and answers of a plurality of roads, and executing the following tasks; the method comprises the following steps:

step S11, removing repeated geometric questions and answers of middle school;

step S12, classifying the geometric questions of middle school and answers into two geometric questions, automatically classifying the geometric questions of middle school with graphics, and classifying the geometric questions of middle school without graphics as geometric questions of middle school without graphics;

step S13, classifying and checking the geometric questions of the middle school, namely manually checking and checking the classification result of the geometric questions of the same middle school;

step S14, after manual inspection and verification, according to the training set: verification set: ratio of test set = 8:1:1, dividing the geometric questions and answers of the middle school;

step S15, after the middle school geometric questions and answers are divided, the middle school geometric questions of the training set and the verification set are manually marked; extracting the problem solving step according to the answer, and marking the problem solving step as a program language which can be identified by a computer in a manual mode.

3. The automatic middle school geometric problem solving method based on contrast learning according to claim 2, wherein the method is characterized in that: the BERT feature encoder in the step S3 is composed of a plurality of layers of bidirectional encoders by using an encoder module in a transducer model framework, and the calculation process is shown in a formula (1);

(1)；

wherein e _i ^w For the ith word token w _i The corresponding word obtained through the BERT feature encoder is embedded into the feature vector.

4. The automatic middle school geometric problem solving method based on contrast learning according to claim 3, wherein the method is characterized in that: the geometric image generator in the step S4 comprises the following specific contents:

s41, inputting data into a geometric image generator, wherein the input data is corresponding word embedded feature vectors in the geometric stem of the non-graphics primitive;

step S42, the geometric image generator is an adapted comparison learning model, the comparison learning model is a classification model of a text-image pair, and the comparison learning model is trained and fine-tuned to achieve a downstream task of generating geometric images;

step S43, training the comparison learning model:

collecting and arranging a graph geometric question data set to obtain a middle school geometric question stem and a corresponding geometric figure;

respectively extracting the characteristics of the geometric stem of middle school and the corresponding geometric figure to obtain a word embedded characteristic vector e _i ^w And geometric feature h ^CNN Forming a text-image pair;

inputting the characteristics of the text-image pairs into a classification model of the text-image pairs for contrast learning, and marking the text-image pairs matched with each other as positive samples and the text-image pairs not matched as negative samples under the condition of manual supervision;

the classification model of the text-image pair can obtain the stem of the middle school geometry and the corresponding geometry through positive samples and negative samples, namely a word embedded feature vector e is given _i ^w Find out the corresponding geometric figure feature h ^CNN ；

Step S44, fine tuning is carried out on the comparison learning model:

defining the text as x, the geometric figure as y, and introducing the generated graphic code into the primary:the calculation process is shown in a formula (2), wherein the graphic code generated by the primary is obtained by training the graphic code of the comparison learning model as a true value;

（2）；

wherein,representing the generation of a geometry y from the text x; />Is expressed as being capable of embedding a feature vector e according to a word after training by a contrast learning model _i ^w To generate geometric feature h ^CNN ； />Representing finding geometric feature h from text x ^CNN Then the geometric feature h ^CNN Decoding to generate a geometric figure y; />Represented as finding geometric feature h from text x ^CNN And the decoded corresponding geometry y, < ->Represented as finding geometric feature h from text x ^CNN ；

Step S45, geometric figure generation loss L _prior : according to the primary in the fine tuning of the contrast learning model, predicting geometric figures by adopting a mean square error loss function, wherein the calculation process is shown in a formula (3);

（3）；

wherein L is _prior The loss of the generation of the geometric figure is represented,representing the sum of the geometric figure generation losses of the previous i times, T being the number of times, h _(i) ^CNN Representing the geometric feature h generated for the ith time ^CNN ；/>Representing the ith generated geometric feature h with text x _(i) ^CNN ；/>Representing the ith generated geometric feature h with text x _(i) ^CNN And geometric figure feature h ^CNN Do bad, wherein->Is an adjustable parameter.

5. The automatic middle school geometric problem solving method based on contrast learning according to claim 4, wherein the method is characterized in that: the step S5 is provided with a graphic problem device which comprises a bidirectional LSTM layer, a graphic encoder, a joint reasoning module and a program decoder; the specific contents include:

step S51, inputting data into a graphic problem solving device, wherein the input data comprises word embedded feature vectors in graphic geometric problem stems obtained by a BERT feature encoder and corresponding geometric figures generated by a non-graphic geometric problem and a geometric image generator;

step S52, bi-directional LSTM layer: word embedded feature vector e in the stem of the geometric question in the diagram obtained by the BERT feature encoder _i ^w Inputting the text into a bidirectional LSTM layer, and acquiring a context semantic feature vector corresponding to the ith word in the mathematical geometry text by utilizing the bidirectional LSTM layer, namely embedding the word into the feature vector e _i ^w Respectively and correspondingly inputting the data into a forward LSTM layer and a backward LSTM layer, as shown in a formula (4);

（4）；

wherein h is _i ^LSTM Context semantic feature vector corresponding to the i-th word in the mathematical geometry text, LSTM _f 、LSTM _b Representing the output vector of the forward LSTM layer and the output vector of the backward LSTM layer respectively,representing a cascading operation;

step S53, the graphics encoder: adopting CNN convolutional neural network mode to realizeExtracting geometric image features h ^CNN The CNN convolutional neural network comprises a convolutional layer, a nonlinear activation function and a pooling layer component; performing convolution operation on the geometric image through sliding convolution check in the convolution layer, and capturing local features; meanwhile, a nonlinear activation function is introduced, the expression capacity of the CNN convolutional neural network is increased, the pooling layer assembly reduces the dimension of the feature map, and key features of geometric figures are reserved; the multi-layer stacked convolution layers enable the CNN convolution neural network to gradually extract geometric features of higher levels; obtaining geometric figure characteristics h through full connection layer ^CNN ；

Step S54, a joint reasoning module: contextual semantic feature vector h corresponding to the ith word through an attention mechanism _i ^LSTM And geometric figure feature h ^CNN Fusion is carried out to realize cross-boundary semantic fusion and alignment, and a context semantic feature vector h corresponding to the ith word containing an attention mechanism is obtained _i ^LSTM And geometric figure feature h ^CNN Multimodal feature vector M corresponding to the ith word of information _i The calculation process is as formula (5) and formula (6);

（5）；

（6）；

where Attention represents the Attention mechanism, Q, K, V represents the query vector, key vector and value vector, respectively, softmax is the normalized exponential function, dd is the second dimension of the query vector Q, key vector K,、/>、/>projection parameter matrixes of a query vector Q, a key vector K and a value vector V corresponding to the ith word in a self-attention mechanism are respectively represented; let->、Wherein->A parameter matrix learned for a linear layer, D representing a transpose;

step S55, program decoder, multi-modal feature vector M _i Feeding the linear layer to obtain an initial state s ₀ Hidden state s of bi-directional LSTM layer at time step t _t Concatenated with the result of interest, feed into the linear layer with Softmax function to predict the next program token P _t Is a distribution of (a).

6. The automatic middle school geometric problem solving method based on contrast learning according to claim 4, wherein the method is characterized in that: generating loss L in step S6 _g Adopting the negative log likelihood of the target program, wherein the calculation formula is shown as (7);

) （7）；

where θ is a parameter of the loss function,is a program token, y _t To generate the target program at time t, y _t-1 To generate the target program at time t-1, M _i Is a multi-modal feature vector.

7. The automatic middle school geometric problem solving method based on contrast learning according to claim 6, wherein the method is characterized in that: the automatic solving model of the geometric problem in middle school is divided into four modules of a BERT feature encoder, a geometric image generator, a graphic problem generator and a unified big model, wherein the BERT feature encoder respectively serially connects the geometric image generator and the graphic problem generator, the geometric image generator and the graphic problem generator are in parallel structures, and then serially unifies the big model.