CN110826649A - End-to-end mathematics blank filling question handwriting recognition system - Google Patents
End-to-end mathematics blank filling question handwriting recognition system Download PDFInfo
- Publication number
- CN110826649A CN110826649A CN201910989935.XA CN201910989935A CN110826649A CN 110826649 A CN110826649 A CN 110826649A CN 201910989935 A CN201910989935 A CN 201910989935A CN 110826649 A CN110826649 A CN 110826649A
- Authority
- CN
- China
- Prior art keywords
- module
- recognition
- recognition system
- mathematical
- gap
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/24—Character recognition characterised by the processing or recognition method
- G06V30/242—Division of the character sequences into groups prior to recognition; Selection of dictionaries
- G06V30/244—Division of the character sequences into groups prior to recognition; Selection of dictionaries using graphical properties, e.g. alphabet type or font
- G06V30/2455—Discrimination between machine-print, hand-print and cursive writing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an end-to-end mathematics filling-in-the-blank handwriting recognition system, which comprises: the acquisition module acquires pictures through a terminal, and performs horizontal correction rotation, distortion reduction and sharpening enhancement on the acquired pictures; the training module trains the acquisition module through a large number of samples; the recognition module is used for performing mixed-row recognition on the Chinese formula collected by the collection module by building a deep learning frame and utilizing a convolutional neural network and a cyclic neural network; the integrated environment and deployment module enables the identification module to be deployed to a cluster network of multiple GPUs through the multiple GPU load balancing module; the system can realize the correction service processing of a large number of high concurrences by setting an integrated environment and a deployment module, in addition, the Chinese formula mixed arrangement recognition is realized through a recognition module, in addition, the acquisition module is trained through a training module, and the mixed recognition of various handwriting styles and formula expression modes and the verification on the validity of an answer area can be realized.
Description
Technical Field
The invention relates to the field of auxiliary education and teaching systems, in particular to a handwriting recognition system for an end-to-end math gap filling question.
Background
At present, the online education industry develops rapidly, and online education companies take teachers, students and parents as original intentions to solve pain points and problems faced in current teaching behaviors. We find that operation correction is a great burden for teachers in the current education and teaching process, and the prior art can say that selected questions are relatively accurately identified and automatically corrected, but the correction process intellectualization of gap filling questions due to unique characteristics faces the following difficulties:
1. the batch correction quantity is large, the concurrency requirement is high, no matter a teacher visits the batch correction service online or a student visits the batch correction service offline, the time is relatively concentrated, and an application scene needs to be fed back to a teacher or a student in a short time
2. In the mathematics teaching behavior, the blank filling questions comprise formulas handwritten by students and Chinese mixed answers, and the handwriting styles of different students are greatly different
3. If students do not normally answer the questions, if the students exceed the answering area, the teachers are difficult to identify the answers and are fuzzy or seriously smeared.
Disclosure of Invention
In order to solve the problems, the invention of the product is provided for creating intelligent gap filling problems. The end-to-end math gap filling question handwriting recognition system comprises a whole set of system from information acquisition to recognition return, and the difficulties mentioned above are fully solved.
In order to achieve the above object, the present invention provides an end-to-end math gap filling question handwriting recognition system, comprising:
the acquisition module acquires pictures through a terminal, and performs horizontal correction rotation, distortion reduction and sharpening enhancement on the acquired pictures;
the training module trains the acquisition module through a large number of samples;
the recognition module is used for performing mixed-row recognition on the Chinese formula collected by the collection module by building a deep learning frame and utilizing a convolutional neural network and a cyclic neural network;
and the integrated environment and deployment module enables the identification module to be deployed to a cluster network with multiple GPUs through the multiple GPU load balancing module.
In the above technical solution, the identification module includes a feature extraction backbone network module, and the feature extraction backbone network module is composed of a plurality of neural network convolution layers, a nonlinear transformation layer, and a pooling layer.
In the above technical solution, the identification module further includes a region proposing network module, configured to perform neural network module learning on the picture features extracted by the feature extraction backbone network module.
In addition, the identification module also comprises an effective area pooling module which calculates an effective image characteristic subgraph by collecting the proposal of the area proposal network and combining the image characteristic graph.
In the above technical solution, the recognition module further includes a generation module, and generates a corresponding preliminary recognition result by performing mixed typesetting expression on the image feature subgraph and the Chinese and mathematical formula Latex of the student answer.
The recognition module also comprises a semantic regularization and completion module, and the error caused by the recognition of the middle module and various writing styles is corrected by performing effectiveness analysis, regularization processing and completion on the primary recognition result.
Preferably, the generating module is generated based on a recurrent neural network.
As a preferred technical solution, further, the acquisition module further includes a labeling module, and the labeling module is configured to label a specific position of the answering area bounding box, an area type, and correct identification content corresponding to the answering.
Compared with the prior art, the invention has the beneficial effects that: the system can realize the correction service processing of a large number of high concurrences by setting an integrated environment and a deployment module, in addition, the Chinese formula mixed arrangement recognition is realized through a recognition module, in addition, the acquisition module is trained through a training module, and the mixed recognition of various handwriting styles and formula expression modes and the verification on the validity of an answer area can be realized.
Drawings
FIG. 1 is a block diagram of the overall structure of the present invention;
FIG. 2 is a structural framework diagram of an identification module;
FIG. 3 is a flow chart of identification module identification;
FIG. 4 is a structural framework diagram of an integrated environment and deployment module.
Detailed Description
The following further describes embodiments of the present invention with reference to the drawings. It should be noted that the description of the embodiments is provided to help understanding of the present invention, but the present invention is not limited thereto. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
As shown in fig. 1, the present invention provides an end-to-end mathematical gap filling question handwriting recognition system, comprising: acquisition module 10, training module 20, recognition module 30, and integrated environment and deployment module 40. The respective modules are explained in detail below.
The acquisition module 10 is used for acquiring a picture of a handwritten mathematical question answering area, a user can use the mobile terminal to shoot, in addition, the acquisition module is provided with an auxiliary line, the user shoots in a guide area through the auxiliary line, the acquisition module is also provided with a correction unit, and the correction unit horizontally corrects and rotates, reduces distortion and enhances sharpening for the picture shot by the user. In addition, the acquisition module also comprises a labeling module which is used for labeling the specific position and the area type of the boundary box of the answering area and correctly identifying the content corresponding to the answering.
The training module 20 is used for training the acquisition module 10, improving the accuracy of acquisition by the acquisition module 10, and acquiring and labeling response samples of a large number of effective users. Through designing and developing a data acquisition module, training data is effectively managed.
The recognition module 30 performs mixed-rank recognition on the Chinese formula collected by the collection module by building a deep learning framework and utilizing a convolutional neural network and a cyclic neural network. As shown in fig. 2 and fig. 3, the recognition module 30 specifically includes a feature extraction backbone network module 31, a region proposal network module 32, an effective region pooling module 33, a generation module 34, and a semantic regularization and completion module 35.
The feature extraction backbone network module 31 is composed of a plurality of neural network convolution layers, a nonlinear transformation layer and a pooling layer, and achieves the standard of high recognition rate through pre-training in a large-scale image classification data set ImageNet.
The area proposal network module 32 performs a neural network module learning through the picture features extracted in the previous step, and outputs a network which is considered as a main area where the handwriting part for filling the blank question exists, wherein the main area is mainly shown in a form of a bounding box, namely four vertex coordinates of a square area. And screening the identified area in the network through a Softmax function, and outputting an area proposal.
The active area pooling module 33 computes an active image feature subgraph to be fed into the subsequent network by collecting the proposal of the area proposal network in combination with the image feature graph.
The generation module 34 performs content-by-content generation on the feature subgraphs of the handwritten image of the effective filling-up question. Each image characteristic subgraph comprises a handwriting part for answering by students filling blank questions. After the neural network architecture obtains the characteristic subgraph, a primary recognition result is generated through a circular neural network structure. Specific examples are: inputting a picture to be recognized into a system, generating a characteristic subgraph by the system through collecting, preprocessing and recognizing parts a-c, and generating Chinese and mathematical formula latex mixed expression from the characteristic subgraph by the generating part through design and training of a cyclic neural network.
The semantic regularization and completion module 35 performs validity analysis, regularization processing and completion on the preliminary recognition result, and corrects errors caused by recognition of a middle module of the system and various writing styles. Specifically, in the semantic recognition process, a previous module generates a primary recognition result expressed by a mixed Chinese and latex arrangement through a characteristic subgraph of a character part and a formula part according to methods such as image segmentation and the like, but the primary recognition result has two small problems, the first part of the recognition result does not conform to the rule for displaying the latex expression, here, a latex expression regularization module is firstly designed and realized to complement the irregularly recognized latex expression, such as the lack of brackets in the expression or the recognition of invalid latex tokens, the second problem is that the primary topological mixed Chinese and mathematical formula part in the recognition result is not fully utilized to context information, a conditional random field model is trained to describe the interdependence probability relationship between the character part and the latex mathematical formula, and the topological mixed arrangement relationship is further regularized through the model, the two modules are used for generating a recognition result with higher accuracy, so that the recognition result presented by the front section is more convenient.
As shown in FIG. 4, the integrated environment and deployment module passes through the multi-GPU load balancing module to enable the identification module to be deployed to a multi-GPU cluster network. The integrated environment module addresses the load pressure brought by high access and high concurrency. A large amount of high-concurrency correction service processing can be realized through the multi-CPU load balancing module.
The embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, and the scope of protection is still within the scope of the invention.
Claims (8)
1. An end-to-end mathematical gap-filling question handwriting recognition system, comprising:
the acquisition module acquires pictures through a terminal, and performs horizontal correction rotation, distortion reduction and sharpening enhancement on the acquired pictures;
the training module trains the acquisition module through a large number of samples;
the recognition module is used for performing mixed-row recognition on the Chinese formula collected by the collection module by building a deep learning frame and utilizing a convolutional neural network and a cyclic neural network;
and the integrated environment and deployment module enables the identification module to be deployed to a cluster network with multiple GPUs through the multiple GPU load balancing module.
2. The end-to-end mathematical gap-filling question handwriting recognition system of claim 1, wherein: the identification module comprises a feature extraction backbone network module, and the feature extraction backbone network module consists of a plurality of neural network convolution layers, a nonlinear transformation layer and a pooling layer.
3. The end-to-end mathematical gap-filling question handwriting recognition system of claim 2, wherein: the identification module also comprises a region proposing network module which is used for learning the picture features extracted by the feature extraction backbone network module by the neural network module.
4. The end-to-end mathematical gap-filling question handwriting recognition system of claim 3, wherein: the identification module also comprises an effective area pooling module which calculates an effective image characteristic subgraph by collecting the proposal of the area proposal network and combining the image characteristic graph.
5. The end-to-end mathematical gap-filling question handwriting recognition system of claim 4, wherein: the recognition module also comprises a generation module which generates a corresponding preliminary recognition result through the image characteristic subgraph and the Chinese and mathematical formula Latex mixed typesetting expression of student answers.
6. The end-to-end mathematical gap-filling question handwriting recognition system of claim 5, wherein: the recognition module also comprises a semantic regularization and completion module, and the error caused by the recognition of the middle module and various writing styles is corrected by performing effectiveness analysis, regularization processing and completion on the primary recognition result.
7. The end-to-end mathematical gap-filling question handwriting recognition system of claim 5, wherein: the generation module is generated based on a recurrent neural network.
8. The end-to-end mathematical gap-filling question handwriting recognition system of claim 1, wherein: the acquisition module further comprises a marking module, and the marking module is used for marking the specific position and the area type of the boundary box of the answering area and correctly identifying the content corresponding to the answering.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910989935.XA CN110826649A (en) | 2019-10-17 | 2019-10-17 | End-to-end mathematics blank filling question handwriting recognition system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910989935.XA CN110826649A (en) | 2019-10-17 | 2019-10-17 | End-to-end mathematics blank filling question handwriting recognition system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110826649A true CN110826649A (en) | 2020-02-21 |
Family
ID=69549475
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910989935.XA Pending CN110826649A (en) | 2019-10-17 | 2019-10-17 | End-to-end mathematics blank filling question handwriting recognition system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110826649A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170206431A1 (en) * | 2016-01-20 | 2017-07-20 | Microsoft Technology Licensing, Llc | Object detection and classification in images |
| CN107169485A (en) * | 2017-03-28 | 2017-09-15 | 北京捷通华声科技股份有限公司 | A kind of method for identifying mathematical formula and device |
| CN110210413A (en) * | 2019-06-04 | 2019-09-06 | 哈尔滨工业大学 | A kind of multidisciplinary paper content detection based on deep learning and identifying system and method |
-
2019
- 2019-10-17 CN CN201910989935.XA patent/CN110826649A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170206431A1 (en) * | 2016-01-20 | 2017-07-20 | Microsoft Technology Licensing, Llc | Object detection and classification in images |
| CN107169485A (en) * | 2017-03-28 | 2017-09-15 | 北京捷通华声科技股份有限公司 | A kind of method for identifying mathematical formula and device |
| CN110210413A (en) * | 2019-06-04 | 2019-09-06 | 哈尔滨工业大学 | A kind of multidisciplinary paper content detection based on deep learning and identifying system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11790641B2 (en) | Answer evaluation method, answer evaluation system, electronic device, and medium | |
| CN109815932B (en) | Test paper correcting method and device, electronic equipment and storage medium | |
| CN109634961B (en) | Test paper sample generation method and device, electronic equipment and storage medium | |
| CN104463101A (en) | Answer recognition method and system for textual test question | |
| CN110619326B (en) | English test paper composition detection and identification system and method based on scanning | |
| CN111652202B (en) | Method and system for solving video question-answer problem by improving video-language representation learning through self-adaptive space-time diagram model | |
| CN107239801A (en) | Video attribute represents that learning method and video text describe automatic generation method | |
| CN105427696A (en) | Method for distinguishing answer to target question | |
| CN107301411B (en) | Mathematical formula identification method and device | |
| CN109242856A (en) | A kind of papery operation electronization reads and makes comments method and device | |
| CN113076900B (en) | Test paper head student information automatic detection method based on deep learning | |
| CN111242024A (en) | Method and system for recognizing legends and characters in drawings based on machine learning | |
| CN111144079B (en) | Method and device for intelligently acquiring learning resources, printer and storage medium | |
| CN110443235B (en) | Intelligent paper test paper total score identification method and system | |
| CN109902683A (en) | Paper core point and Data Input method and system | |
| CN113159014A (en) | Objective question reading method, device, equipment and storage medium based on handwritten question numbers | |
| CN110929582A (en) | Automatic correction method and device for oral calculation questions, storage medium and electronic equipment | |
| CN112446259A (en) | Image processing method, device, terminal and computer readable storage medium | |
| CN112116840B (en) | Job correction method and system based on image recognition and intelligent terminal | |
| CN111986117A (en) | System and method for correcting arithmetic operation | |
| CN111008594A (en) | Error correction evaluation method, related equipment and readable storage medium | |
| CN112926571A (en) | Formula picture recognition and question judgment system based on deep learning | |
| CN117095414A (en) | Handwriting recognition system and recognition method based on dot matrix paper pen | |
| JP7293658B2 (en) | Information processing device, information processing method and program | |
| CN110826649A (en) | End-to-end mathematics blank filling question handwriting recognition system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |