KR102508200B1 - Artificial intelligence learning data collection method for improving recognition rate using eraser function and candidate list - Google Patents

Abstract

In the present specification, an AI (Artificial Intelligence) processor receives an image containing a mathematical expression input by the user from a user device in a method of collecting learning data from a user in order to learn an AI model, and the user inputs Based on an image including an equation: using the AI model, 1) a character string of the equation and 2) a candidate list related to the character string are predicted, and 1) the character string of the equation and 2 are predicted by the user device. ) transmitting a candidate list related to the character string; From the user device, learning data generated based on the character string of the equation and the candidate list may be received, and the AI model may be trained using the learning data.

Description

Artificial intelligence learning data collection method for improving recognition rate using eraser function and candidate list}

The present specification relates to a method and an apparatus for constructing a learning model with a higher recognition rate by effectively collecting error data that occurs when reading characters input by a user's typeface or stroke method in an artificial intelligence model.

AI learning data is a generic term for data used for learning AI models such as machine learning and deep learning. In an artificial intelligence model for reading characters input by a user's typeface or stroke method, existing artificial intelligence learning data collection is collected through artificial actions of developers.

In addition, artificial intelligence learning data sets are initially collected in an artificial way, and when commercial services are in progress, data of users using services are randomly collected and used for additional learning without distinguishing between data that is essential for function improvement and data that is not. there was a problem with

Publication No. 10-2016-0101683 (20160825) Publication No. 10-2016-0018495 (20160217) Publication No. 10-2018-0060971 (20180607)

The purpose of the present specification is to naturally connect a series of actions that occur in the process of reaching a recognition result desired by a user inputting a character, that is, a process of correcting an erroneously recognized character to a learning data collection step, thereby effectively improving the recognition rate. We propose a method for securing high-purity data (eg, raw data for learning and correct answer data).

In addition, the purpose of this specification is to propose a data collection method that can expect better performance even with little additional learning by effectively securing data of parts that are not well recognized in the already learned artificial intelligence model.

The technical problems to be achieved by this specification are not limited to the above-mentioned technical problems, and other technical problems not mentioned are clear to those skilled in the art from the detailed description of the specification below. will be understandable.

One aspect of the present specification is a method of collecting learning data from a user so that an AI (Artificial Intelligence) processor learns an AI model, comprising the steps of receiving an image including a mathematical expression input by the user from a user device. ; Based on the image including the mathematical expression input by the user: predicting 1) a character string of the mathematical expression and 2) a candidate list related to the character string using the AI model; transmitting to the user device 1) a character string of the equation and 2) a candidate list related to the character string; receiving, from the user device, learning data generated based on the character string of the equation and the candidate list; and learning the AI model using the learning data.

In addition, the learning data may be composed of only a pair of a correct answer candidate selected by the user from the candidate list through the user device and a string of the mathematical expression corresponding to the correct answer candidate.

In addition, the user device 1) displays an icon for erasing and re-entering a specific character of the character string of the equation on the display unit, 2) when the icon is selected by the user, the user erases it from the user 3) the character the user wants to erase is not displayed on the display unit, 4) the user inputs a character that replaces the character the user wants to erase, 5) the replacement A text may be displayed at a location where a text the user wants to erase was displayed.

In addition, the learning data may be composed of only a pair of the text to be erased by the user and the text to be replaced.

Another aspect of the present specification is a mathematical recognition device for collecting learning data from a user in order to learn an AI (Artificial Intelligence) model, comprising: a memory containing the AI model; A communication module for transmitting and receiving a signal by being connected to a user device; and a processor controlling the memory and the communication module, wherein the processor receives an image including a mathematical formula input by the user from the user device through the communication module, and receives an image including a mathematical formula input by the user, and Based on the image including the expression: Using the AI model, 1) a string of the equation and 2) a candidate list related to the string are predicted, and the user device receives 1) the string of the equation and 2) the Transmits a candidate list related to a character string, receives learning data generated based on the character string of the equation and the candidate list from the user device, and uses the learning data to learn the AI model. .

In addition, another aspect of the present specification is a method of collecting learning data from a user in order for a user device to learn an AI (Artificial Intelligence) model, comprising: receiving an input of a mathematical expression from the user through a display unit; Transmitting the image including the mathematical expression to a mathematical recognition device; receiving, from the math recognition device, a character string of the equation predicted by the equation recognition device based on the image including the equation; displaying the character string of the mathematical expression to the user through the display unit; erasing a specific character from the character string of the equation on the display unit and displaying an icon for re-inputting; receiving, from the user, a character to be erased by the user when the icon is selected; receiving a text that replaces the text the user wants to erase without displaying the text the user wants to erase on the display unit; displaying the replacement text at the position where the text the user wants to erase was displayed; and transmitting, to the mathematical expression recognition device, learning data for learning the generated AI model based on the character to be replaced. can include

In addition, the learning data may be composed of only a pair of a letter to be erased by the user and a letter to be replaced.

According to the embodiments of the present specification, by collecting artificial intelligence learning data through a use process for reaching a recognition result performed by a user, it is possible to efficiently secure high-purity data effective for improving the recognition rate.

In addition, according to an embodiment of the present specification, by proposing a method for effectively learning data of a part that is not well recognized in an already learned artificial intelligence model, better performance of the artificial intelligence model can be expected with little additional learning. .

Effects obtainable in the present specification are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. .

1 is a diagram for conceptually explaining the configuration of a system for recognizing mathematical expressions of the present specification.
2 is a diagram for explaining the configuration of a mathematical expression recognition device according to an embodiment of the present specification.
3 is an example of a DNN model to which the present specification can be applied.
4 is an example of a candidate recommendation method in general text recognition that can be applied in this specification.
5 and 6 are examples of a display unit to which the present specification can be applied.
7 is an example of a conventional mathematical expression recognition method.
8 is an example of a learning method through a candidate list to which this specification can be applied.
9 is an example of a learning method through an eraser function to which the present specification can be applied.
10 is an embodiment to which the present specification may be applied.
11 is an embodiment of an AI processor to which the present specification can be applied.
The accompanying drawings, which are included as part of the detailed description to aid understanding of the present specification, provide examples of the present specification and describe technical features of the present specification together with the detailed description.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of this specification , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

1 is a diagram for conceptually explaining the configuration of a system for recognizing mathematical expressions of the present specification.

Referring to FIG. 1 , a system for recognizing a mathematical expression may include a user device 100 and a mathematical expression recognition device 300 .

The user device 100 may access the mathematical expression recognition device 300 using a communication network. A user may input and receive a mathematical expression by accessing the mathematical expression recognition device 300 using the user device 100 and transmitting/receiving data with the mathematical expression recognition device 300 .

The user device 100 is an IP-assigned device and can perform network communication through the Internet or the like. The user device 100 may be a wireless communication device that ensures portability and mobility. For example, the user device 100 includes navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT ( International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smart pad, All kinds of hand held based wireless communication devices such as tablet PCs, desktop PCs, slate PCs, notebook(laptop) computers, PMP (Portable Multimedia Player) and the like may be included.

It is natural that the use device 100 to which this specification can be applied is not limited to the above-described types, and may include all types of devices capable of communicating with external devices.

The above-described user device 100 may install or mount an app for recognizing mathematical formulas. The user may receive a service for recognizing a mathematical expression described later from the mathematical recognition device 300 by accessing the mathematical expression recognition device 300 through an app installed in the user device 100 .

The mathematical expression recognition device 300 may read mathematical expressions input to the user device 100 connected through a communication network. The mathematical expression recognition device 300 may include a processor 310, a memory 330, a communication module 350, and the like.

The processor 310 may include hardware/software that performs language processing, and may control driving of software and input/output and functions of the mathematical expression recognition device 300 .

The memory 330 may store data as a pre-built knowledge base and store various databases related to machine learning. The knowledge base may include various types of information for reading a mathematical expression input from a user.

The communication module 330 may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS or NFC.

Details of the mathematical expression recognition device 300 will be described in detail with reference to FIG. 2 .

2 is a diagram for explaining the configuration of a mathematical expression recognition device according to an embodiment of the present specification.

Referring to FIG. 2 , a device 300 for recognizing a mathematical expression according to an embodiment of the present specification may include a processor 310, a memory 330, and a communication module 350.

The processor 310 may include one or more application processors (APs), one or more communication processors (CPs), or one or more AI processors (artificial intelligence processors). The application processor, communication processor, or AI processor may be each included in different integrated circuit (IC) packages or included in one IC package.

The application processor may control a plurality of hardware or software components connected to the application processor by driving an operating system or an application program, and may process/calculate various data including multimedia data. For example, the application processor may be implemented as a system on chip (SoC). The processor 310 may further include a graphic processing unit (GPU).

The communication processor may perform a function of managing a data link and converting a communication protocol in communication with the user device 100 connected to the network. For example, the communication processor may be implemented as an SoC. The communications processor may perform at least part of the multimedia control function.

Also, the communication processor may control data transmission and reception of the communication module 350 . The communication processor may be implemented to be included as at least a part of the application processor.

The application processor or communication processor may load a command or data received from at least one of a non-volatile memory or other components connected thereto into the volatile memory and process the load. Also, the application processor or communication processor may store data received from at least one of the other components or generated by at least one of the other components in a non-volatile memory.

The memory 330 may include a built-in memory or an external memory. The built-in memory includes volatile memory (eg, dynamic RAM (DRAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), etc.) or non-volatile memory (eg, one time programmable ROM (OTPROM)), It may include at least one of programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, NAND flash memory, NOR flash memory, etc.). According to one example, the embedded memory may take the form of a solid state drive (SSD). The external memory is a flash drive, for example, compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), or It may include a memory stick and the like.

The communication module 350 may include a wireless communication module or an RF module. The wireless communication module may include, for example, Wi-Fi, BT, GPS or NFC. For example, the wireless communication module may provide a wireless communication function using a radio frequency. Additionally or alternatively, the wireless communication module includes a network interface or modem for connecting the user device 100 to a network (eg, Internet, LAN, WAN, telecommunication network, cellular network, satellite network, POTS or 5G network, etc.) can include

The RF module may be responsible for transmitting and receiving data, for example, transmitting and receiving RF signals or called electronic signals. For example, the RF module may include a transceiver, a power amp module (PAM), a frequency filter, or a low noise amplifier (LNA). In addition, the RF module may include components for transmitting and receiving electromagnetic waves in free space in wireless communication, for example, conductors or wires.

The processor 310 may include an AI processor 311, a data learning unit 311a, a data preprocessing unit 311b, a data selection unit 311c, a model evaluation unit 311d, a response module 315, and the like. .

The AI processor 311 may learn a neural network using a program stored in the memory 330 . In particular, the AI processor 311 may learn a neural network for recognizing mathematical formulas input from the user devices 100 . Here, the neural network may be designed to simulate a human brain structure (eg, a neuron structure of a human neural network) on a computer. A neural network may include an input layer, an output layer, and at least one hidden layer. Each layer may include at least one neuron having a weight, and the neural network may include neurons and synapses connecting the neurons. In the neural network, each neuron may output an input signal input through a synapse as a function value of an activation function for weight and/or bias.

A plurality of network modes may transmit and receive data according to a connection relationship, respectively, so as to simulate synaptic activity of neurons that transmit and receive signals through synapses. Here, the neural network may include a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes may exchange data according to a convolutional connection relationship while being located in different layers. Examples of neural network models include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks, restricted Boltzmann machines, and deep belief networks. ), and deep Q-Network, and can be applied to fields such as vision recognition, voice recognition, natural language processing, and voice/signal processing.

Meanwhile, the processor 310 performing the functions described above may be a general-purpose processor (eg, CPU), or may be an AI-only processor (eg, GPU) for artificial intelligence learning.

The memory 330 may store various programs and data necessary for the operation of the user device 100 and/or the mathematical expression recognition device 300 . The memory 330 is accessed by the AI processor 311, and reading/writing/modifying/deleting/updating of data by the AI processor 311 may be performed. In addition, the memory 330 may store a neural network model (eg, a deep learning model) generated through a learning algorithm for data classification/recognition according to an embodiment of the present specification. Furthermore, the memory 330 may store not only the learning model 221 but also input data, learning data, learning history, and the like.

Meanwhile, the AI processor 311 may include a data learning unit 311a that learns a neural network for data classification/recognition. The data learning unit 311a may learn criteria for which training data to use to determine data classification/recognition and how to classify and recognize data using the training data. The data learning unit 311a may acquire learning data to be used for learning and learn the deep learning model by applying the obtained learning data to the deep learning model.

The data learning unit 311a may be manufactured in the form of at least one hardware chip and mounted in the mathematical expression recognition device 300 . For example, the data learning unit 311a may be manufactured in the form of a dedicated hardware chip for artificial intelligence, and may be manufactured as a part of a general-purpose processor (CPU) or a graphics-only processor (GPU) to be installed in the mathematical recognition device 300. can Also, the data learning unit 311a may be implemented as a software module. When implemented as a software module (or a program module including instructions), the software module may be stored in a computer-readable, non-transitory computer readable recording medium (non-transitory computer readable media). In this case, at least one software module may be provided to an operating system (OS) or may be provided by an application.

The data learning unit 311a may learn to have a criterion for determining how to classify/recognize predetermined data by using the obtained training data. At this time, the learning method by the data learning unit 311a may be classified into supervised learning, unsupervised learning, and reinforcement learning. Here, supervised learning refers to a method of learning an artificial neural network given a label for training data, and a label is an answer (or a result value) that the artificial neural network must infer when learning data is input to the artificial neural network. can mean Unsupervised learning may refer to a method of training an artificial neural network in a state in which a label for training data is not given. Reinforcement learning may refer to a method of learning to select an action or an action sequence that maximizes a cumulative reward in each state by an agent defined in a specific environment. In addition, the data learning unit 311a may train the neural network model using a learning algorithm including backpropagation or gradient descent. When the neural network model is learned, the learned neural network model may be referred to as a learning model 331 . The learning model 331 may be stored in the memory 330 and used to infer results for new input data other than learning data.

On the other hand, the AI processor 311 is a data preprocessing unit 311b and/or a data selection unit in order to improve analysis results using the learning model 331 or to save resources or time required for generating the learning model 331. (311c) may be further included.

The data pre-processing unit 311b may pre-process the acquired data so that the acquired data can be used for learning/reasoning for situation determination. For example, the data preprocessing unit 311b may extract feature information as preprocessing for input data acquired through an input device, and the feature information may include a feature vector, a feature point, or It can be extracted in a format such as a feature map.

The data selector 311c may select data necessary for learning from among the learning data preprocessed by the data learner 311a or the data preprocessor 311b. The selected training data may be provided to the learning model 331 . For example, the data selection unit 311c may select only data about an object included in a mathematical formula determined to be “correct” as training data based on a mathematical formula input by a user through voice, text, or various input devices. there is. Also, the data selector 311c may select data necessary for reasoning from among input data obtained through an input device or input data preprocessed by a preprocessor.

In addition, the AI processor 311 may further include a model evaluation unit 311d to improve the analysis result of the neural network model. The model evaluation unit 311d inputs evaluation data to the neural network model, and when an analysis result output from the evaluation data does not satisfy a predetermined criterion, it may cause the model learning unit to learn again. In this case, the evaluation data may be preset data for evaluating the learning model 331 . For example, the model evaluator 311d determines whether a predetermined criterion is not satisfied when the number or ratio of evaluation data for which the analysis result is not accurate among the analysis results of the learned neural network model for the evaluation data exceeds a preset threshold. can be evaluated as

The response module 315 may respond to the candidate of the mathematical expression input to the user device under the control of the processor. A detailed description of this will be described later.

The communication module 350 may transmit the AI processing result by the AI processor 311 to the user device 100 .

The user device 100 may also include the above-described structure of the mathematical recognition device 300 .

Deep Neural Network (DNN) model

3 is an example of a DNN model to which the present specification can be applied.

A deep neural network (DNN) is an artificial neural network (ANN) composed of several hidden layers between an input layer and an output layer. Deep neural networks can model complex non-linear relationships, just like regular artificial neural networks.

For example, in a deep neural network structure for an object identification model, each object may be represented as a hierarchical composition of basic image elements. In this case, the additional layers may consolidate the characteristics of the gradually gathered lower layers. This feature of deep neural networks allows complex data to be modeled with fewer units (units, nodes) compared to similarly performed artificial neural networks.

As the number of hidden layers increases, the artificial neural network is called “deep.” The machine learning paradigm that uses a sufficiently deep artificial neural network as a learning model is called deep learning. In addition, a sufficiently deep artificial neural network used for such deep learning is collectively referred to as a deep neural network (DNN).

In this specification, the artificial neural network used for this deep learning method in the AI processor 311 and / or learning model 331 is collectively referred to as DNN, but if meaningful data can be output in a similar way, other methods It goes without saying that a deep learning method can be applied.

4 is an example of a candidate recommendation method in general text recognition that can be applied in this specification.

Referring to FIG. 4 , a user may input text through the display unit 410 included in the user device 100 . For example, the display unit 410 may implement a touch screen by forming a mutual layer structure or integrally with a touch sensor. Such a touch screen may function as an input unit providing an input interface between the user device 100 and the user, and may provide an output interface between the user device 100 and the user.

The display unit 410 may output text input by the user to the user. The user device 100 may receive a PenSize associated with the thickness of an input character from the user.

The user device 100 may receive the candidate list 420 of the character input by the user from the mathematical expression recognition device 300 and display the candidate list 420 . For example, when the user writes 'Kanada' on the display unit 410, the mathematical expression recognition device 300 uses the learning model 331 to write 'Kanada', 'Kanana', 'Kanaa', etc. An expected candidate list may be generated and transmitted to the user device 100 . The user can select a suitable candidate from the list of candidates displayed on the display unit 410 .

However, it is difficult to apply the generally used method to mathematical expression recognition as it is. For example, a mathematical expression may consist of one or two short numbers and letters, but may also consist of complex structures other than letters, such as fractions, subscripts, and upper and lower combinations. In addition, when an equation having such a complicated combination of structures is input to the user device 100, a problem in that the number of cases in the candidate list generated by the learning model 331 is too large may occur.

5 and 6 are examples of a display unit to which the present specification can be applied.

Referring to FIGS. 5 and 6 , the user device 100 may receive a candidate list for each element of the input equation from the equation recognition device 300 in order to solve the above problem.

For example, the equation recognition device 300 predicts each element of the equation by using the learning model 331 for the image including the equation received from the user device 100, and generates a candidate list of each element. can create

Referring to FIG. 5 ( a ) , a user may input a mathematical expression into the user device 100 . For example, a user may input a mathematical expression through the display unit 410 .

The user device 100 may display the input mathematical formula and transmit an image including the input mathematical formula to the mathematical recognition device 300 . The equation recognition device 300 analyzes the image including the equation to determine 1) a character string of the equation and 2) a candidate list of elements included in the equation string, and transmits the result to the user device 100. there is.

For example, the string of equations may be a set of elements predicted by the learning model 331 with the highest accuracy.

Also, the candidate list may be a list including elements predicted by the learning model 331 and having accuracy within a certain range.

Referring to FIG. 5( b ), the user device 100 may display the received mathematical expression string. However, even in this case, the user intends and inputs the equation of FIG. ) can be passed on.

Referring to FIGS. 6(a), 6(b) and 6(c) , when a user selects an erroneous character in a character string of a mathematical expression, the user device 100 provides a candidate related to the character selected from the candidate list. can display them.

The user may select a correct answer candidate representing an intended letter from among displayed candidates, and the user device 100 may replace the erroneous letter with the selected correct answer candidate and display a mathematical expression including the replaced letter.

Referring to FIG. 6(d), if there is an error in the structure of a part of a mathematical formula rather than an error in individual characters, the user can request re-recognition of the mathematical formula by erasing only the error and re-entering the error.

For example, when a user selects an eraser icon and a character to be erased is selected, the user device 100 may not display the selected character, and the user inputs a replacement character into the user device 100. can do. The user device 100 may display the input replacement text at the location where the selected text was displayed. In addition, the user device 100 may transmit the regenerated image of the mathematical formula to the mathematical recognition device 300 again, and request the aforementioned mathematical recognition procedure again.

Of course, the above-described operation of the mathematical recognition device 300 can be performed as a similar operation in the user device 100 itself in an offline state through an app installed in the user device 100 . To this end, the app may include a learning model 331.

7 is an example of a conventional mathematical expression recognition method.

Referring to FIG. 7, in the case of conventional optical recognition and stroke mathematical recognition (for example, MS math app), when modifying the mathematical expression input by the user, the first input handwriting is corrected with an eraser and re-recognized, It was a method of erasing the incorrect part of the input result and requesting correction again.

In this way, when recognizing the mathematical expression again, the user device 100 needs to request the artificial intelligence server or the application performing the corresponding function to perform the same task as the first time. Since this artificial intelligence recognition requires a lot of computer resources, a waste of cost was incurred by going through several recognition processes in order to completely recognize one formula.

However, the method presented in this specification does not incur additional cost because recognition can be completed through a candidate list delivered together with the initial prediction result in most cases.

In addition, through a process of approving correct text through a list of candidates performed by a user, the artificial intelligence model can obtain artificial intelligence learning data without additional cost.

In addition, since there is no re-recognition process, correction of the mathematical formula can be performed immediately, so that the speed of inputting the mathematical formula can be increased.

In addition, since the above-described eraser function directly corrects the result of the recognized mathematical expression differently from the conventional method, it is possible to maintain the normally recognized character as it is and to re-recognize only the corrected part.

More specifically, since the recognition rate of the typed portion is much higher than that of the other portion, it may be advantageous to modify the structure of the mathematical expression and to be recognized again.

In addition, in the artificial intelligence model, normally output letters can be expected to be output again, but in the case of modified letters using the eraser function, the artificial intelligence model is likely to judge the wrong result again. , Learning data for characters with a low recognition rate can be effectively secured through the acquisition of artificial intelligence learning data of modified characters using the eraser function.

8 is an example of a learning method through a candidate list to which this specification can be applied, and FIG. 9 is an example of a learning method through an eraser function to which this specification can be applied.

Referring to FIGS. 8 and 9 , the user device 100 and the AI processor 311 may additionally perform a learning method for obtaining artificial intelligence learning data in addition to the operations of FIGS. 5 and 6 described above.

Referring to FIG. 8 , the initial learning model 331 may have a learned state using artificial intelligence learning data collected through a developer.

Referring to FIG. 8( a ) , the user device 100 may receive a mathematical expression from the user in a typeface or stroke method.

Referring to FIG. 8( b ), the AI processor 311 may receive an image of a mathematical expression from the user device 100 and deliver the recognized result to the user device 100 through the learning model 331. there is.

Referring to FIG. 8(c), the user selects a symbol of the corresponding character in order to correct the incorrectly recognized character, the user device 100 outputs a candidate list of the selected symbol, and the user selects a desired character from the candidate list. can choose

Referring to FIG. 8(e), if the candidate list includes the correct text, the user can select the correct text from the candidate list, and the user device 100 replaces the erroneously recognized text with the correct text, can display In this case, in order to train the learning model 331, the user device 100 may make a pair of the correct text and raw data input by the first user, store them, and transmit them to the AI processor 311. The AI processor 311 may train the learning model 331 using the received data pair. In this case, 1) the user device 100 transfers the entire formula input by the user and all correct answer characters in pairs, or 2) transfers only the pair of the incorrectly recognized character and the correct answer character, so that the user device 100 It is possible to reduce the signaling cost between the AI processor 311.

Additionally, in the following cases, the user device 100 may not perform an operation of transmitting the data pair to the AI processor 311 by determining that the raw data is not in a good state and therefore inappropriate as training data:

(1) If the user is not satisfied with the final result and does not perform the final approval step

(2) When structural formulas that are not symbols (for example, formulas containing structures such as fractions and sigma) are deleted

(3) In case of modifying more than 2 symbols in one modification

(4) If there is only additional formula input by the user without the act of deleting the symbol

Referring to FIG. 8(d), if the candidate list does not include the correct answer text, the user may not select the correct answer text from the candidate list. In this case, the user device 100 may prepare to perform the eraser function by displaying the symbol input by the first user.

Referring to FIGS. 9(a) to 9(d) , the user may select an eraser icon and re-enter only a specific character (eg, re-enter the character f as 5). When the user is satisfied with the result newly presented by the user device 100 and performs an approval step (eg, a step of utilizing the result such as inserting a document), the user device 100 is configured to train the learning model 331. , The correct answer text and the original data input by the first user can be paired and stored, and transmitted to the AI processor 311. The AI processor 311 may train the learning model 331 using the received data pair. Even in this case, 1) the user device 100 transfers the entire formula input by the user and all correct answer characters in pairs, or 2) transfers only the pair of the incorrectly recognized character and the correct answer character, so that the user device 100 ) and the signaling cost between the AI processor 311 can be reduced.

In addition, the user device 100 determines that the raw data is not in a good state and is inappropriate as learning data in the following cases, in the same manner as in the above step, and does not perform an operation of transmitting the data pair to the AI processor 311. may not:

(1) If the user is not satisfied with the final result and does not perform the final approval step

(2) When structural formulas that are not symbols (for example, formulas containing structures such as fractions and sigma) are deleted

(3) In case of modifying more than 2 symbols in one modification

(4) If there is only additional formula input by the user without the act of deleting the symbol

10 is an embodiment to which the present specification may be applied.

Referring to FIG. 10 , the user device 100 is connected to the AI processor 311 through a network and may include an app for recognizing a mathematical expression.

The user device 100 receives a mathematical expression from the user through the display unit 410 (S1010).

The user device 100 transmits the image including the mathematical expression to the AI processor 311 through a network (S1020).

The AI processor 311 uses the learning model 331 to predict a string of equations based on an image including the equations (S1030). For example, the AI processor 311 may predict a string of mathematical expressions through OCR (optical character recognition) using the learning model 331 . In more detail, the AI processor 311 may use the learning model 331 to predict the character string of the mathematical expression as a set of characters having the highest reliability value.

The AI processor 311 uses the learning model 331 to predict a candidate list of each element included in the string of the predicted equation (S1040). For example, the candidate list may be a set of characters having a certain range of reliability values.

The AI processor 311 transmits the mathematical string and the candidate list to the user device 100 through the network (S1050).

The user device 100 displays a string of mathematical formulas on the display unit 410 (S1060).

The user device 100 performs error correction of equations through the user and stores learning data for learning the learning model 331 (S1070).

For example, the user device 100 may receive an erroneous character selected from a character string of a mathematical expression from a user, and display one or more candidates related to the erroneous character in a candidate list. In more detail, the user device 100 may display the candidates by sorting them according to reliability values of the candidates. Thereafter, the user device 100 may select a correct candidate from among displayed candidates from the user. The user device 100 may replace the erroneous character with the selected correct answer candidate and display the result on the display unit 410 .

In this case, in order to train the learning model 331, the user device 100 may make a pair of correct answer text and raw data related to a mathematical formula input by the first user and store them.

In addition, the user device 100 may display an eraser icon for erasing and re-entering a specific character in a mathematical string. When the eraser icon is selected by the user, the user device 100 may receive selection of a character the user wants to erase from the user and may not display the character the user wants to erase on the display unit 410 .

The user device 100 may receive a text to replace the text the user wants to erase from the user, and display the text to be replaced at the location where the text the user wants to erase was displayed. For example, the user device 100 may re-execute steps S1020 to S1060 in order to re-determine the alternative character through the installed app or receive a prediction through the learning model 331.

Even in this case, the user device 100 may pair and store the answer text and raw data input by the first user in order to train the learning model 331 .

The user device 100 transfers the learning data to the AI processor 311 (S1080). For example, the user device 100 may transfer a correct answer candidate selected by the user or a character input by the user to the AI processor 311 in order to train the learning model 331 . In this case, 1) the user device 100 transmits the entire formula input by the user and all correct answer characters in pairs, or 2) transmits only the pair of the incorrectly recognized element and the correct answer element, so that the user device 100 It is possible to reduce the signaling cost between the AI processor 311.

The AI processor 311 re-learns the learning model 331 using the learning data (S1090). Through this, the learning model 331 can efficiently collect data for learning from the user device 100 in various usage environments.

11 is an embodiment of an AI processor to which the present specification can be applied.

Referring to FIG. 11, the operation of the AI processor 311 of FIG. 10 is illustrated in more detail. The AI model may include a learning model 331 .

1. The AI processor 311 receives an image including a mathematical formula input by the user from a user device. For example, the user device 100 may receive a mathematical expression from the user through the display unit 410 . The user device 100 may transmit an image including a mathematical expression to the AI processor 311 through a network.

2. The AI processor 311 predicts 1) a character string of the equation and 2) a candidate list related to the character string using the connected AI model based on the image including the equation input by the user.

3. The AI processor 311 transmits 1) a string of equations and 2) a candidate list related to the string to the user device. For example, the user device 100 may receive, from the AI processor 311, a character string of the equation predicted in an AI model based on an image including the equation, and a candidate list related to the character string of the equation. can The user device 100 may display a mathematical expression string to the user through the display unit 410, and may select one or more erroneous characters from the mathematical expression string from the user. In addition, the user device 100 displays at least one or more candidates related to an erroneous character on the display unit 410 based on a candidate list related to a character string of a mathematical formula, and displays the one or more candidates from the user. Among them, the correct candidate is selected. For example, the candidate for the correct answer may mean a character intended by the user among the candidates. In addition, the user device 100 may replace the erroneous character with the correct answer candidate and display it on the display unit 410 . In addition, the user device 100 may display an icon for erasing a specific character from the character string of the mathematical expression on the display unit 410 and inputting it again. The icon may mean the aforementioned eraser icon. If the icon is selected by the user, the user device 100 may receive a selection of text that the user wants to erase from the user. The user device 100 may not display the text the user wants to erase on the display unit 410 and may receive a text that replaces the text the user wants to erase from the user. The user device 100 may display the input replacement text at the location where the text the user wants to erase was displayed. Through this, the user device 100 may generate an image including a mathematical formula in which an error in the mathematical formula is corrected. The user device 100 may transmit the image including the corrected mathematical expression to the AI processor 311 . For example, the corrected mathematical expression may include replacement characters input by the user.

4. The AI processor 311 receives learning data generated from the user device based on the character string of the equation and the candidate list. For example, the user device 100, based on the image including the corrected equation from the AI processor 311, 1) a string of the corrected equation predicted by the AI model, and 2) the corrected equation. A candidate list related to the mathematical expression string may be received, and the corrected mathematical expression string may be displayed to the user through the display unit 410 . Through this, the string of the predicted corrected equation can be expected to have higher reliability than the string of the first equation.

5. The AI processor 311 may learn the AI model using the learning data. Such learning data consists of only a pair of a correct answer candidate selected from the candidate list by the user through the user device and a string of the mathematical expression corresponding to the correct answer candidate, or through an eraser icon, the user wants to erase the learning data. It can consist only of a pair of letters and the above replacing statements.

The above specification can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. , and also includes those implemented in the form of a carrier wave (eg, transmission over the Internet). Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of this specification should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of this specification are included in the scope of this specification.

In addition, although services and embodiments have been described above, this is only an example and does not limit the present specification, and those skilled in the art to which this specification belongs will not deviate from the essential characteristics of the present service and embodiments. It will be appreciated that various modifications and applications not exemplified above are possible. For example, each component specifically shown in the embodiments can be modified and implemented. And differences related to these modifications and applications should be construed as being included in the scope of the present specification as defined in the appended claims.

Claims (11)

delete delete delete delete delete delete delete delete In a method for collecting learning data from a user in order for a user device to learn an AI (Artificial Intelligence) model,
receiving an input of a mathematical expression from the user through a display unit;
Transmitting the image including the mathematical expression to a mathematical recognition device;
Receiving, from the equation recognition device, a character string of the equation predicted by the equation recognition device based on an image including the equation, and a candidate list related to the character string of the equation;
displaying the character string of the mathematical expression to the user through the display unit;
receiving, from the user, a character with an error selected from the text string of the mathematical expression;
displaying on the display unit one or more candidates related to the erroneous character based on the candidate list;
Based on the user's selection of the correct answer letter from among the candidates:
generating first learning data for learning the AI model; and
transmitting the first learning data to the mathematical expression recognition device;
Based on not receiving a selection of the correct answer letter from the user:
erasing a specific character from the character string of the equation on the display unit and displaying an icon for re-inputting;
receiving, from the user, a character to be erased by the user when the icon is selected;
receiving a text that replaces the text the user wants to erase without displaying the text the user wants to erase on the display unit;
displaying the replacement text at the position where the text the user wants to erase was displayed;
generating second learning data for learning the AI model based on the text to be replaced; and
transmitting the first learning data or the second learning data to the mathematical expression recognition device;
Including,
The first learning data is
1) the erroneous character, and 2) a pair of the correct answer character,
The second learning data is
Learning data collection method, consisting of a pair of 1) the character the user wants to erase, and 2) the replacement character.
delete In a user device that collects learning data from a user in order to learn an AI (Artificial Intelligence) model,
a communication module for communicating with a mathematical expression recognition device;
display unit; and
a processor for functionally controlling the communication module and the display unit; Including,
The processor
Receiving an input of a mathematical expression from the user through the display unit;
Transmitting the image containing the equation to the equation recognition device,
Receiving, from the equation recognition device, a character string of the equation predicted by the equation recognition device based on an image including the equation, and a candidate list related to the character string of the equation;
Displaying the character string of the mathematical expression to the user through the display unit;
The user selects an erroneous character among the strings of the equation,
Based on the candidate list, at least one or more candidates related to the erroneous character are displayed on the display unit;
Based on the user's selection of the correct answer letter from among the candidates:
Generating first learning data for learning the AI model, transmitting the first learning data to the equation recognition device,
Based on not receiving a selection of the correct answer letter from the user:
Erase a specific character from the character string of the equation on the display unit and display an icon for re-inputting;
When the icon is selected, the user selects a character to be erased,
Receiving input from the user of a character replacing the character the user wants to erase without displaying the character the user wants to erase on the display unit;
Display the replacement character at the position where the character the user wants to erase was displayed;
Based on the replacement character, second learning data for learning the AI model is generated;
Transmitting the first learning data or the second learning data to the mathematical expression recognition device;
The first learning data is composed of a pair of 1) the erroneous character and 2) the correct answer character,
The second learning data is composed of a pair of 1) a character to be erased by the user, and 2) a character to be replaced.

Images