KR102241480B1 - Methods and devices for correcting reading errors in mathematical OCR(Optical Character Recognition) and stroke recognition - Google Patents

Abstract

The present specification provides a method for alleviating reading errors in mathematical equation recognition. The method for recognizing a mathematical equation inputted from a user by a user device comprises: a step of a receiving a first mathematical equation through a display unit from the user; a step of transmitting an image including the first mathematical equation to a network; a step of receiving a string of the first mathematical equation predicted in the network and a candidate list associated with the string of the first mathematical equation based on the image including the first mathematical equation from the network; a step of displaying the string of the first mathematical equation through the display unit to the user; a step of receiving a selection for a character with an error in the string of the first mathematical equation from the user; a step of displaying one or more candidates associated with the character with an error on the display unit based on the candidate list associated with the string of the first mathematical equation; and a step of receiving a selection for a correct answer candidate among the one or more candidates from the user.

Description

[Methods and devices for correcting reading errors in mathematical OCR (Optical Character Recognition) and stroke recognition]

In this specification, when recognizing an equation written in a handwriting and print image or device stroke method using a learning technology, if accurate reading is not possible due to the characteristics of the character even if the recognition rate is increased, the user who received the result of the reading is It relates to a method and apparatus for easily and quickly modifying an equation.

Optical character recognition (OCR) is the conversion of an image of a character written by a human or printed by a machine with an image scanner and converted into machine-readable characters. It is a software that converts the printed image of a document that can be obtained by scanning an image into a format such as a character code that can be edited by a computer, and is generally referred to as OCR, and OCR began as a research field of artificial intelligence or machine vision.

" 중 무엇인지 구분하는 것에도, 어려움이 있다.In the case of recognizing an equation using the conventional OCR, for example, when writing as 0.x, it may be very ambiguous as to which of the three "0" to be recognized as "0, o, o". In addition, it can be ambiguous whether "." is also a decimal point, a multiplication, or sometimes a comma. Also, "x" is also "

There is also a difficulty in distinguishing which one is among ".

Unexamined Patent Publication No. 10-2016-0101683 (20160825) Unexamined Patent Publication No. 10-2016-0018495 (20160217) Unexamined Patent Publication No. 10-2018-0060971 (20180607)

An object of the present specification is to propose a method for improving the reading error of mathematical expression recognition.

In addition, an object of the present specification is to propose a method of delivering a candidate list together with a prediction result of an initial equation to a user in order to improve a reading error in recognition of an equation.

The technical problems to be achieved by the present specification are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are obvious to those of ordinary skill in the technical field to which the present specification belongs from the detailed description of the following specification. It will be understandable.

An aspect of the present specification is a method of recognizing an equation input from a user in a user device, the method comprising: receiving a first equation from the user through a display unit; Transmitting an image including the first equation to a network; Receiving, from the network, 1) a character string of the first equation, and 2) a candidate list related to the character string of the first equation, predicted by the network, based on the image including the first equation; Displaying the character string of the first equation to the user through the display unit; Receiving, from the user, a character with an error in the character string of the first equation; Displaying at least one or more candidates related to the erroneous character on the display unit based on a candidate list related to the character string of the first equation; And receiving, from the user, selecting a correct answer candidate from among the one or more candidates.

In addition, replacing the errored character with the correct answer candidate and displaying it on the display unit; It may further include.

In addition, deleting a specific character from the character string of the first equation on the display unit and displaying an icon for inputting again; When the icon is selected, receiving, from the user, a character to be erased by the user; Not displaying the character to be erased by the user on the display unit, and receiving a character to replace the character to be erased by the user from the user; And displaying the replacement character at a position where the character to be erased by the user has been displayed. It may further include.

In addition, transmitting an image including a second equation to the network, wherein the second equation includes the replacement character; Receiving, from the network, a list of candidates related to the character string of 1) the second equation, and 2) the character string of the second equation, predicted by the network, based on the image including the second equation; And displaying the character string of the second equation to the user through the display unit.

Another aspect of the present specification is a user device for recognizing an equation input from a user, comprising: a display unit; A transceiver for transmitting and receiving signals; And a processor that functionally controls the display unit and the transceiver, wherein the processor receives a first equation from the user through the display unit, and the image including the first equation through the transceiver Is transmitted to the network, and from the network, based on the image including the first equation, 1) the character string of the first equation, and 2) the candidate related to the character string of the first equation, predicted by the network Receiving a list, displaying the character string of the first equation to the user through the display unit, and receiving a selection of a character with an error from the character string of the first equation from the user, and the first equation Based on the candidate list related to the character string of, at least one or more candidates related to the erroneous character may be displayed on the display unit, and a correct answer candidate may be selected from the one or more candidates from the user.

Another aspect of the present specification is a method for a network to recognize an equation input from a user through a user device, the method comprising: receiving, from the user device, an image including an equation input from the user; Predicting the character string of the equation based on the image and the learning model for predicting the character string of the equation, and generating a candidate list related to the predicted character string of the equation; And transmitting the character string of the equation and the candidate list to the user device. It may include.

In addition, receiving, from the user device, a result of correcting an error in the character string of the equation; And retraining the learning model by using the result value.

According to an embodiment of the present specification, it is possible to improve a reading error in recognition of an equation.

In addition, according to an exemplary embodiment of the present specification, in order to improve a reading error in recognition of an equation, a list of candidates together with a prediction result of the initial equation may be delivered to the user.

In addition, according to the exemplary embodiment of the present specification, since it is possible to edit an equation including reading error correction on one screen without separating the input unit and the result unit, the utilization of the device screen space is increased.

The effects obtainable in the present specification are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description. .

1 is a diagram conceptually illustrating a configuration of a system for recognizing an equation of the present specification.
2 is a diagram illustrating a configuration of an equation recognition device according to an embodiment of the present specification.
3 is an example of an OCR model to which the present specification can be applied.
4 is an example of a candidate recommendation method in general character recognition that can be applied in the present specification.
5 and 6 are examples of a display unit to which the present specification can be applied.
7 is an example of an existing method for recognizing an equation.
8 is an embodiment to which the present specification may be applied.
9 is an embodiment of a user device to which the present specification may be applied.
The accompanying drawings, which are included as part of the detailed description to aid in understanding of the present specification, provide embodiments of the present specification, and describe technical features of the present specification together with the detailed description.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are assigned the same reference numerals regardless of the reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "unit" for constituent elements used in the following description are given or used interchangeably in consideration of only the ease of preparation of the specification, and do not have meanings or roles that are distinguished from each other by themselves. In addition, in describing the embodiments disclosed in the present specification, when it is determined that a detailed description of related known technologies may obscure the subject matter of the embodiments disclosed in the present specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are for easy understanding of the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present specification It should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first and second may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

Singular expressions include plural expressions unless the context clearly indicates otherwise.

In this application, terms such as "comprises" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance.

1 is a diagram conceptually illustrating a configuration of a system for recognizing an equation of the present specification.

Referring to FIG. 1, a system for recognizing an equation may include a user device 100 and an equation recognition device 300.

The user device 100 may access the equation recognition device 300 using a communication network. A user may input and read an equation by accessing the equation recognition device 300 using the user device 100 and transmitting and receiving data with the equation recognition device 300.

The user device 100 is an IP-allocated device and may perform network communication through the Internet or the like. The user device 100 may be a wireless communication device in which portability and mobility are guaranteed. For example, the user device 100 includes navigation, Personal Communication System (PCS), Global System for Mobile communications (GSM), Personal Digital Cellular (PDC), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), and IMT ( International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smart pad, All kinds of hand held wireless communication devices such as tablet PC, desktop PC, slate PC, notebook computer, It may include a Portable Multimedia Player (PMP) or the like.

It is natural that the use device 100 to which the present specification is applicable is not limited to the above-described types, and may include all types of devices capable of communicating with external devices.

The user device 100 described above may install or mount an app for recognizing an equation. By accessing the equation recognition device 300 through an app installed on the user device 100, the user may receive a service for equation recognition, which will be described later, from the equation recognition device 300.

The equation recognition device 300 may read an equation input to the user device 100 connected through a communication network. The equation recognition device 300 may include a processor 310, a memory 330, a communication module 350, and the like.

The processor 310 includes hardware/software that performs language processing, and may control the operation of software and input/output and functions of the mathematical expression recognition device 300.

The memory 330 may store data in a pre-built knowledge base and store various databases related to machine learning. The knowledge base may include various pieces of information for reading an equation input from a user.

The communication module 330 may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS or NFC.

Details of the equation recognition device 300 will be described in detail with reference to FIG. 2.

2 is a diagram illustrating a configuration of an equation recognition device according to an embodiment of the present specification.

Referring to FIG. 2, a device 300 for recognizing an equation according to an embodiment of the present specification may include a processor 310, a memory 330, and a communication module 350.

The processor 310 may include one or more application processors (APs), one or more communication processors (CP), or at least one or more artificial intelligence processors (AI processors). The application processor, the communication processor, or the AI processor may be included in different integrated circuit (IC) packages, respectively, or may be included in one IC package.

The application processor may drive an operating system or an application program to control a plurality of hardware or software components connected to the application processor, and perform various data processing/operations including multimedia data. For example, the application processor may be implemented as a system on chip (SoC). The processor 310 may further include a graphic processing unit (GPU).

The communication processor may perform a function of managing a data link and converting a communication protocol in communication with the user device 100 connected through a network. For example, the communication processor may be implemented as an SoC. The communication processor may perform at least a part of the multimedia control function.

In addition, the communication processor may control data transmission/reception of the communication module 350. The communication processor may be implemented to be included as at least a part of the application processor.

The application processor or the communication processor may load and process commands or data received from at least one of a nonvolatile memory or other components connected to each of the volatile memory. In addition, the application processor or the communication processor may store data received from at least one of the other components or generated by at least one of the other components in the nonvolatile memory.

The memory 330 may include an internal memory or an external memory. The built-in memory is a volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.)) or a non-volatile memory non-volatile memory (e.g., one time programmable ROM (OTPROM), It may include at least one of PROM (programmable ROM), EPROM (erasable and programmable ROM), EEPROM (electrically erasable and programmable ROM), mask ROM, flash ROM, NAND flash memory, NOR flash memory, etc.). According to an example, the internal memory may take the form of a solid state drive (SSD). The external memory is a flash drive, for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), or It may include a memory stick or the like.

The communication module 350 may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS or NFC. For example, the wireless communication module may provide a wireless communication function using a radio frequency. Additionally or alternatively, the wireless communication module provides a network interface or modem for connecting the user device 100 to a network (eg, Internet, LAN, WAN, telecommunication network, cellular network, satellite network, POTS or 5G network, etc.). Can include.

The RF module may be responsible for transmitting and receiving data, for example, transmitting and receiving an RF signal or a called electronic signal. For example, the RF module may include a transceiver, a power amp module (PAM), a frequency filter, or a low noise amplifier (LNA). In addition, the RF module may include a component for transmitting and receiving an electromagnetic wave in a free space in wireless communication, for example, a conductor or a conducting wire.

The processor 310 may include an AI processor 311, a data learning unit 311a, a data preprocessing unit 311b, a data selection unit 311c, a model evaluation unit 311d, a response module 315, and the like. .

The AI processor 311 may learn a neural network using a program stored in the memory 330. In particular, the AI processor 311 may learn a neural network for recognizing an equation input from between the user devices 100. Here, the neural network may be designed to simulate a human brain structure (eg, a neuron structure of a human neural network) on a computer. The neural network may include an input layer, an output layer, and at least one hidden layer. Each layer includes at least one neuron having a weight, and the neural network may include a synapse connecting neurons and neurons. In a neural network, each neuron may output an input signal input through a synapse as a function value of an activation function for weight and/or bias.

The plurality of network modes can send and receive data according to their respective connection relationships to simulate the synaptic activity of neurons that send and receive signals through synapses. Here, the neural network may include a deep learning model developed from a neural network model. In a deep learning model, a plurality of network nodes are located in different layers and can exchange data according to a convolutional connection relationship. Examples of neural network models include a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network, a restricted Boltzmann machine, and a deep belief network. ), and deep Q-Network, and can be applied in fields such as vision recognition, speech recognition, natural language processing, and voice/signal processing.

Meanwhile, the processor 310 performing the above-described function may be a general-purpose processor (eg, a CPU), but may be an AI-only processor (eg, a GPU) for artificial intelligence learning.

The memory 330 may store various programs and data necessary for the operation of the user device 100 and/or the equation recognition device 300. The memory 330 is accessed by the AI processor 311, and data read/write/edit/delete/update by the AI processor 311 may be performed. In addition, the memory 330 may store a neural network model (eg, a deep learning model) generated through a learning algorithm for classifying/recognizing data according to an embodiment of the present specification. Furthermore, the memory 330 may store not only the learning model 221 but also input data, training data, and learning history.

Meanwhile, the AI processor 311 may include a data learning unit 311a that learns a neural network for data classification/recognition. The data learning unit 311a may learn a criterion for how to classify and recognize data using which training data to use in order to determine data classification/recognition. The data learning unit 311a may learn the deep learning model by acquiring training data to be used for training and applying the acquired training data to the deep learning model.

The data learning unit 311a may be manufactured in the form of at least one hardware chip and mounted on the equation recognition device 300. As an example, the data learning unit 311a may be manufactured in the form of a dedicated hardware chip for artificial intelligence, and may be manufactured as a part of a general-purpose processor (CPU) or a graphics dedicated processor (GPU) to be mounted on the equation recognition device 300. I can. Also, the data learning unit 311a may be implemented as a software module. When implemented as a software module (or a program module including an instruction), the software module may be stored in a computer-readable non-transitory computer readable media. In this case, at least one software module may be provided to an operating system (OS) or may be provided by an application.

The data learning unit 311a may learn to have a criterion for determining how the neural network model classifies/recognizes predetermined data by using the acquired training data. In this case, the learning method by the data learning unit 311a may be classified into supervised learning, unsupervised learning, and reinforcement learning. Here, supervised learning refers to a method of training an artificial neural network when a label for training data is given, and the label is the correct answer (or result value) that the artificial neural network must infer when training data is input to the artificial neural network. Can mean Unsupervised learning may mean a method of training an artificial neural network in a state in which a label for training data is not given. Reinforcement learning may mean a method in which an agent defined in a specific environment learns to select an action or action sequence that maximizes the cumulative reward in each state. In addition, the data learning unit 311a may train the neural network model using a learning algorithm including an error backpropagation method or a gradient decent method. When the neural network model is trained, the trained neural network model may be referred to as a learning model 331. The learning model 331 may be stored in the memory 330 and used to infer a result of new input data other than the training data.

Meanwhile, the AI processor 311 is a data preprocessing unit 311b and/or a data selection unit to improve the analysis result using the learning model 331 or to save resources or time required for generating the learning model 331 (311c) may be further included.

The data preprocessor 311b may preprocess the acquired data so that the acquired data can be used for learning/inference for determining a situation. As an example, the data preprocessor 311b may extract feature information as a preprocess for input data acquired through an input device, and the feature information may be a feature vector, a feature point, or It can be extracted in a format such as a feature map.

The data selection unit 311c may select data necessary for learning from among the training data preprocessed by the data learning unit 311a or the data preprocessing unit 311b. The selected training data may be provided to the learning model 331. As an example, the data selection unit 311c may select only data for an object included in the equation determined as the "correct answer" as the learning data, based on a user's voice, text, or an equation input through various input devices. have. In addition, the data selection unit 311c may select data necessary for inference among input data obtained through an input device or input data preprocessed by a preprocessor.

In addition, the AI processor 311 may further include a model evaluation unit 311d to improve the analysis result of the neural network model. The model evaluation unit 311d may input evaluation data to the neural network model, and when an analysis result output from the evaluation data does not satisfy a predetermined criterion, the model learning unit may retrain. In this case, the evaluation data may be preset data for evaluating the learning model 331. As an example, the model evaluation unit 311d does not satisfy a predetermined criterion when the number or ratio of evaluation data in which the analysis result is not accurate among the analysis results of the learned neural network model for evaluation data exceeds a preset threshold. It can be evaluated as

The response module 315 may respond to a candidate of an equation input to the user device under the control of the processor. A detailed description of this will be described later.

The communication module 350 may transmit the AI processing result by the AI processor 311 to the user device 100.

The user device 100 may also include the configuration of the above-described equation recognition device 300.

Deep learning-based OCR (Optical Character Recognition) model

3 is an example of an OCR model to which the present specification can be applied.

The OCR model is an automatic recognition technology that converts text and images on printed materials or photographs into digital data. Examples of technology use include recognition of text on business cards or handwriting information on documents. The general OCR model operates as a subdivided module, such as a module that searches for a text line and a module that divides a letter. The feature that recognizes these different patterns of characters had to be designed by the developer himself. In addition, limited operation was possible only in high-quality images.

In recent years, the accuracy of the OCR field has been improved by applying deep learning, and through learning a large amount of data, it creates a feature extraction for text recognition from images by itself. The following is an example of an OCR model using this deep learning technology.

S71: pre-processing

Computers can recognize pixels with similar brightness values as chunks, so characters with different colors from their surroundings and with a different structure of consecutive dots can be detected more easily. For this, the recognition rate can be greatly improved by going through a pre-processing process.

An example of this pretreatment process is as follows. Converts low color images to grayscale. Then, histogram equalization is performed. By redistributing the brightness distribution of the image to maximize the contrast, you can have a sharper image. However, there is still a limit to clearly distinguishing between background and text. To solve this, we proceed with binarization. If the pixel value is 255 (white), it is changed to '0', and if the pixel value is 0 to 254 (gray and black), it is changed to '1'. As a result, the background and text can be separated more clearly.

S72: text detecting

After putting the image into a learning model 331 such as a DNN, feature values are obtained. The data to be obtained are the text box and the rotation angle of the text area. Choosing a text area from the input image can reduce unnecessary calculations. Rotation information is used when making an inclined text area horizontal. Then crop the image into text units. After this step, an individual character image or word image can be obtained.

S73: text recognition

In order to recognize which characters are contained in each cropped image, a learning model is used. The learning model learns how to recognize individual words and characters in the form of images. On the other hand, the types of words or strings that the learning model can recognize differs for each language. Therefore, for general-purpose OCR, it is also necessary to have a module that estimates language using only images.

S74: post-processing

OCR also post-processes character recognition errors in a manner similar to how humans accept text. There are two main methods. The first is how to use the characteristics of each letter. It corrects errors by distinguishing letters (similar pairs) of similar forms such as'work/mil','myo/yang', and'pa/ta'. The second is a method of using contextual information. For this, a language model or dictionary is required, and a language model that has learned a lot of text data on the web is also built through deep learning.

The object of the present specification is to apply an existing deep learning-based OCR model in a more advanced form through a system for mathematical expression recognition. To this end, the user device 100 and/or the equation recognition device 300 may include an OCR model.

4 is an example of a candidate recommendation method in general character recognition that can be applied in the present specification.

Referring to FIG. 4, a user may input text through the display unit 410 included in the user device 100. For example, the display unit 410 may form a layer structure with a touch sensor or are integrally formed to implement a touch screen. Such a touch screen may function as an input unit providing an input interface between the user device 100 and the user, and may provide an output interface between the user device 100 and the user.

The display unit 410 may output a character input by the user to the user. The user device 100 may receive a PenSize associated with the thickness of the input character from the user. In addition, the user device 100 may receive a recognizable language selected from the user through OCR.

The user device 100 may receive and display a candidate list 420 of a character input by the user from the equation recognition device 300. For example, when the user writes'Kanada' on the display unit 410, the equation recognition device 300 uses the learning model 331,'Kanada','Kanaa','Kanaa', etc. An expected candidate list may be generated and transmitted to the user device 100. The user may select a suitable candidate from the candidate list displayed on the display unit 410.

However, it is difficult to apply the generally used method to mathematical expression recognition as it is. For example, the equation may be composed of one or two short numbers and letters, but may be composed of a complex structure other than letters, such as fractions, subscripts, and upper and lower combinations. In addition, when the user device 100 inputs an equation having a combination of such a complex structure, there may be a problem in that the number of candidate lists generated by the learning model 331 is too large.

5 and 6 are examples of a display unit to which the present specification can be applied.

5 and 6, in order to solve the above-described problem, the user device 100 may receive a list of candidates for each element of an input equation from the equation recognition device 300.

For example, the equation recognition device 300 predicts each element of the equation through OCR using the learning model 331 on the image containing the equation received from the user device 100, and is a candidate for each element. You can create a list.

Referring to FIG. 5A, a user may input an equation into the user device 100. For example, the user may input an equation through the display unit 410.

The user device 100 may display the input equation and may transmit an image including the input equation to the equation recognition device 300. The equation recognition device 300 analyzes the image containing the equation, determines 1) a character string of the equation and 2) a candidate list of each element included in the character string of the equation, and transmits it to the user device 100. have.

For example, the string of the equation may be a set of elements with the highest accuracy predicted by the learning model 331.

Also, the candidate list may be a list including elements predicted by the learning model 331 and having a certain range of accuracy.

Referring to FIG. 5B, the user device 100 may display a received string of equations. However, even in this case, the user intended and input the equation of FIG. 5(c), but the learning model 331 erroneously predicts the “0” element as the “o” or “O” element, and the user device 100 ).

6(a), 6(b), and 6(c), when a user selects a character with an error from the character string of an equation, the user device 100 is a candidate related to the character selected from the candidate list. Can be displayed.

The user may select a correct answer candidate representing the intended character from among the displayed candidates, and the user device 100 may replace the erroneous character with the selected correct answer candidate and display an equation including the replaced character.

Referring to FIG. 6(d), if there is an error in some equation structure rather than an individual character error, the user may request re-recognition of the equation by deleting only the error and re-entering it.

For example, when the user selects an eraser icon and the user selects a character to be erased, the user device 100 may not display the selected character, and the user inputs a replacement character into the user device 100 can do. The user device 100 may display the input replacement character at a location where the selected character has been displayed. In addition, the user device 100 may transmit the regenerated image of the equation again to the equation recognition device 300 to request the above-described equation recognition procedure again.

It goes without saying that the above-described operation of the equation recognition device 300 may be performed by a similar operation in the user device 100 itself in an offline state through an app installed in the user device 100. To this end, the app is capable of mathematical expression OCR, and may include a learning model 331.

7 is an example of an existing method for recognizing an equation.

Referring to FIG. 7, in the case of conventional optical recognition and stroke mathematical recognition (eg, MS math app), when a user inputs a mathematical expression, the first handwritten handwriting is corrected with an eraser and re-recognized, It was a method of erasing the wrong part of the input result and requesting correction again.

In this manner, when the equation is re-recognized, the user device 100 must again request the artificial intelligence server or an application that performs the function to perform the same operation as the first time. Since such artificial intelligence recognition requires a lot of computer resources, there was a waste of cost of going through the recognition process several times in order to fully recognize one formula. In addition, in this method, the handwriting input space is insufficient because the input window and the result window must be separated.

However, most of the schemes presented in the present specification do not incur additional costs since recognition can be completed through a candidate list delivered together with the initial prediction result.

In addition, since there is no re-recognition process, the correction of the equation can be performed immediately, and the speed of inputting the equation can be increased.

In addition, since the above-described eraser function directly corrects the result of the recognized equation differently from the conventional method, it is possible to exert an effect of maintaining the normally recognized character as it is and re-recognizing only the modified part.

In more detail, since the recognition rate of the printed part is much higher than that of the handwriting of the non-printed part, it may be advantageous to correct the structure of the equation and receive it again.

In addition, the method presented in the present specification is advantageous in implementing a wide input window even on a device with a small screen, since the input window and the result window can be implemented on one screen.

8 is an embodiment to which the present specification may be applied.

Referring to FIG. 8, the user device 100 is connected to a network and may include an app for recognizing an equation. The network includes an equation recognition device 300.

The user device 100 receives a first equation from the user through the display unit 410 (S810).

The user device 100 transmits an image including the first equation to the network (S820).

The network predicts the character string of the first equation based on the image including the first equation (S830). For example, the network may predict the character string of the first equation through OCR using the learning model 331. In more detail, the network may predict the character string of the first equation using the learning model 331 as a set of characters having the highest reliability value.

The network predicts a candidate list of each element included in the predicted character string of the first equation (S840). For example, the candidate list may be a set of characters having a certain range of reliability values.

The network transmits the character string and the candidate list of the first equation to the user device 100 (S850).

The user device 100 displays the character string of the first equation on the display unit 410 (S860).

The user device 100 performs error correction of the first equation by the user (S870).

For example, the user device 100 may receive a selection of a character with an error from the character string of the first equation from the user, and display one or more candidates related to the character with an error from the candidate list. In more detail, the user device 100 may display candidates by sorting them according to their confidence values. Thereafter, the user device 100 may receive a correct answer candidate selected from among the displayed candidates from the user. The user device 100 may replace a character with an error with a selected correct answer candidate and display it on the display unit 410.

In addition, the user device 100 may display an eraser icon for erasing and re-entering a specific character from the character string of the first equation. When the eraser icon is selected by the user, the user device 100 may receive a character to be erased by the user from the user and may not display the character to be erased by the user on the display unit 410.

The user device 100 may receive a character that replaces a character that the user wants to erase from the user, and display the character that the user wants to erase at a location where the character that the user wants to erase is displayed.

The user device 100 transmits the error correction result value to the network (S880). For example, in order to train the learning model 331, the user device 100 may transmit a correct answer candidate selected by the user or a replacement character input by the user to the network.

The network retrains the learning model 331 using the error correction result value (S890). Through this, the network may receive data for learning from the user device 100 in various use environments.

9 is an embodiment of a user device to which the present specification may be applied.

Referring to FIG. 9, the operation of the user device 100 of FIG. 8 is illustrated in more detail.

1. The user device 100 receives a first equation from a user through the display unit 410.

2. The user device 100 transmits an image including the first equation to the network.

3. The user device 100 from the network, based on the image including the first equation, predicted in the network, 1) the character string of the first equation, and 2) related to the character string of the first equation Receive a list of candidates.

4. The user device 100 displays the character string of the first equation to the user through the display unit 410, and displays one or more characters with errors in the character string of the first equation from the user. Be chosen.

5. The user device 100 displays at least one or more candidates related to the erroneous character on the display unit 410 based on the candidate list related to the character string of the first equation, and from the user, the From one or more candidates, a candidate with a correct answer is selected. For example, the correct answer candidate may mean a character intended by the user among candidates.

In addition, the user device 100 may replace the erroneous character with the correct answer candidate and display it on the display unit 410.

In addition, the user device 100 may erase a specific character from the character string of the first equation on the display unit 410 and display an icon for inputting it again. The icon may mean the above-described eraser icon.

If the icon is selected by the user, the user device 100 may receive a character to be erased by the user from the user.

The user device 100 may not display a character that the user wants to erase on the display unit 410, and may receive a character that replaces the character that the user wants to erase from the user.

The user device 100 may display the input replacement character at a location where the character desired to be erased by the user has been displayed.

Through this, the user device 100 may generate an image including the second equation in which the error in the first equation is corrected. The user device 100 may transmit an image including the second equation to the network. For example, the second equation may include a replacement character input by the user.

The user device 100 is a list of candidates related to the 1) the second equation predicted by the network, and 2) the second equation, based on the image including the second equation from the network. After receiving, the user can display the character string of the second equation through the display unit 410.

Through this, the predicted character string of the second equation can be expected to have a higher reliability than the character string of the first equation.

The foregoing specification can be implemented as computer-readable code on a medium in which a program is recorded. The computer-readable medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid state disks (SSDs), silicon disk drives (SDDs), ROMs, RAM, CD-ROMs, magnetic tapes, floppy disks, optical data storage devices, etc. There is also a carrier wave (for example, transmission over the Internet) also includes the implementation of the form. Therefore, the detailed description above should not be construed as restrictive in all respects and should be considered as illustrative. The scope of this specification should be determined by reasonable interpretation of the appended claims, and all changes within the scope of equivalents of this specification are included in the scope of this specification.

In addition, although the services and embodiments have been described above, these are only examples and do not limit the specification, and those of ordinary skill in the field to which this specification belongs will not depart from the essential characteristics of the service and embodiments It will be appreciated that various modifications and applications not illustrated above are possible. For example, each component specifically shown in the embodiments can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present specification defined in the appended claims.

Claims (10)

In a user device, a method of recognizing an equation input from a user,
Receiving a first equation from the user through a display unit;
Transmitting an image including the first equation to a network;
Receiving, from the network, 1) a character string of the first equation, and 2) a candidate list related to the character string of the first equation, predicted by the network, based on the image including the first equation;
Displaying the character string of the first equation to the user through the display unit;
Receiving, from the user, a character with an error in the character string of the first equation;
Displaying at least one or more candidates related to the erroneous character on the display unit based on a candidate list related to the character string of the first equation;
Receiving, from the user, a candidate for a correct answer from among the one or more candidates;
Replacing the erroneous character with the correct answer candidate and displaying it on the display unit;
Erasing a specific character from the character string of the first equation on the display unit and displaying an icon for inputting it again;
When the icon is selected, receiving, from the user, a character to be erased by the user;
Not displaying the character to be erased by the user on the display unit, and receiving a character to replace the character to be erased by the user from the user; And
Displaying the text to be replaced at a position where the text to be erased by the user has been displayed;
Including, a method of recognizing an equation input from a user.
delete delete The method of claim 1,
Transmitting an image including a second equation to the network, wherein the second equation includes the replacement character;
Receiving, from the network, a list of candidates related to the character string of 1) the second equation, and 2) the character string of the second equation, predicted by the network, based on the image including the second equation; And
Displaying the character string of the second equation to the user through the display unit;
A method for recognizing an equation input from a user further comprising a.
In a user device for recognizing an equation input from a user,
A display unit;
A transceiver for transmitting and receiving signals; And
Includes; a processor that functionally controls the display unit and the transceiver,
The processor is
Receiving a first equation from the user through the display unit,
Through the transceiver, the image including the first equation is transmitted to a network, and from the network, based on the image including the first equation, 1) the character string of the first equation, predicted by the network, And 2) receiving a list of candidates related to the character string of the first equation,
Displaying the character string of the first equation to the user through the display unit,
From the user, a character with an error is selected from the character string of the first equation,
Displaying at least one or more candidates related to the erroneous character on the display unit based on the candidate list related to the character string of the first equation,
From the user, from among the one or more candidates, a correct answer candidate is selected,
The errored character is replaced with the correct answer candidate and displayed on the display unit,
Delete a specific character from the character string of the first equation on the display unit and display an icon for re-entry,
When the icon is selected, a character to be erased by the user is selected from the user,
The user does not display a character to be erased on the display unit, and a character to replace the character to be erased by the user is input by the user,
A user device for recognizing an equation input from a user, displaying the replacement character at a position where the character desired to be erased by the user has been displayed.
delete delete The method of claim 5,
The processor is
Through the transceiver, an image including a second equation is transmitted to the network, and the second equation includes the replacement character,
1) a character string of the second equation predicted by the network, and 2) a candidate list related to the character string of the second equation, based on the image including the second equation, from the network,
A user device for recognizing an equation input from a user to display the character string of the second equation to the user through the display unit.

delete delete

Images