KR101989960B1 - Real-time handwriting recognition method using plurality of machine learning models, computer-readable medium having a program recorded therein for executing the same and real-time handwriting recognition system - Google Patents

Abstract

Provided is a real-time handwriting recognition method using a plurality of machine learning models, which comprises the steps of: utilizing a data set including a plurality of handwriting samples generating a plurality of different types of machine learning models by a server; inputting handwriting on a terminal; selecting one among the plurality of machine learning models; and using the selected machine learning model to recognize the handwriting as text. Accordingly, when handwriting is inputted on the terminal, the handwriting can be quickly and accurately converted into text data regardless of the overlapping degree of the handwriting.

Description

TECHNICAL FIELD [0001] The present invention relates to a real-time handwriting recognition method using a plurality of machine learning models, a computer-readable recording medium storing a program for executing the same, and a real- and a real-time handwriting recognition system.

The present invention relates to a real time handwriting recognition method using a plurality of machine learning models, a computer readable recording medium on which a program for executing the same is recorded, and a real time handwriting recognition system, and more particularly, The present invention relates to a technology that can be recognized separately.

The conventional handwriting recognition program has a limit in that it can be recognized only when handwriting is entered neatly. Especially, when handwriting is mistakenly written or mistakenly overlapped, the handwriting recognition program often fails to correctly recognize the input value.

When using a device such as a smart phone or a tablet PC, the display is generally slippery than paper, making it difficult to input neatly when entering handwriting on the display. As a result, you may inadvertently overwrite the text, or you can write more negative than usual.

When the text is input in this manner, the handwriting recognition program can recognize the handwriting with a character different from that intended by the user. That is, the conventional handwriting recognition program is inconvenient to recognize the overlapping handwriting or the handwriting, so that the user may not input the handwriting as intended and repeatedly input the handwriting again .

In addition, when the machine learning model is introduced to improve accuracy, it takes 1 second to recognize a single letter according to the input state of the letter, and it may take 10 seconds or longer if it is long.

According to Korean Patent Laid-Open No. 10-2016-0003112 (Patent Document 1), a method, a system, and a computer-readable medium related to a real-time handwriting recognition management technique are provided. More particularly, the handwriting recognition module is trained to have a repertoire that can recognize tens of thousands of characters using a single handwriting recognition model and include a number of non-overlapping scripts. At this time, training the multi-script handwriting recognition model is provided as a single convolutional neural network.

Accordingly, the characters of at least three non-overlapping scripts displayed in the multi-script training corpus using the convolutional neural network can be distinguished, but only if the text is not used to overlap each other.

For example, if a user quickly handwritten a series of numbers "123-4567" on the handset, the numbers "1", "2" and "3" could be superimposed on each other. In such a case, according to the prior art, the overlapped numbers can not be recognized.

In addition, the handwriting recognition model can be trained in the same manner as the convolution neural network, but the recognition speed is slow for the numbers superimposed on each other, which makes it difficult to recognize in real time.

Accordingly, when the user quickly inputs a series of numbers into a handwritten form on the handset, it is inconvenient to pay attention to the interval between the numbers. In addition, when numbers are superimposed on a fast input, it takes a long time to recognize them as text, and there is a problem that trial and error occur and re-input is required.

Therefore, nowadays, there is a need for a model capable of quickly recognizing handwritten characters that are severely overlapped.

KR 10-2016-0003112 A

In order to solve the above-mentioned problems, the present invention provides a real-time recognition method capable of quickly recognizing overlapping handwriting text, a computer-readable recording medium recording a program for executing the same, and a real- .

Other objects and advantages of the present invention will become apparent from the following description and the appended claims. It will also be readily apparent that the objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

In order to achieve the above object, a real-time handwriting recognition method using a plurality of machine learning models according to the present invention comprises: (a) a data set including a plurality of handwriting samples is used by a server to generate a plurality of different types of machine learning models ; (b) inputting handwriting on the terminal; (c) applying the plurality of machine learning models; (d) selecting any one of the plurality of machine learning models; And (e) the selected one machine learning model is used to recognize the handwriting as text.

In addition, in the step (d), the selected one machine learning model preferably has the fastest result output speed among the plurality of machine learning models.

In the step (d), it is preferable that the selected one machine learning model has the highest accuracy among the plurality of machine learning models.

It is preferable that the application of the remaining machine learning model is stopped when the one machine learning model is selected in the step (d) while the plurality of machine learning models are applied in the step (c).

In addition, the plurality of machine learning models preferably include at least one of Regression, Convolutional Neural Network (CNN), ResNet, and CapsNet.

(F) further storing and updating the handwriting and the text in the data set; And (g) by the server, the updated data set is utilized to generate a plurality of different types of machine learning models.

A computer readable recording medium according to the present invention for achieving the above object is a computer readable recording medium on which a program for executing a real time handwriting recognition method using a plurality of machine learning models is recorded.

According to an aspect of the present invention, there is provided a system for real-time handwriting recognition using a plurality of machine learning models, the system comprising: a handwriting input terminal; And a server capable of communicating with the terminal, wherein the server comprises: a memory for storing a plurality of machine learning models, a data set, and the plurality of machine learning models, wherein the one of the plurality of machine learning models Is selected and used so that the handwriting is recognized as text.

It is further preferred that the apparatus further comprises a translator and a stream portion including a plurality of streams connected to the translator, wherein one machine learning model is generated from each of the plurality of streams.

In addition, the translator is preferably a Java virtual machine.

In addition, the plurality of machine learning models preferably include at least one of regression, convolutional neural networks, resynet, and capsnet.

The details of other embodiments are included in the detailed description and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and / or features of the present invention and the manner of achieving them will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. And is provided to fully explain the scope of the present invention to those skilled in the art.

As described above, according to the present invention, when handwriting is input on a terminal, it can be quickly and accurately converted into text data regardless of the overlapping degree of handwriting.

Also, even if the numbers are completely overlapped on the handset, they can be converted into text data and used as text data on the terminal.

FIG. 1 is a flowchart for explaining a main configuration of a real-time recognition method using a plurality of machine learning models according to a preferred embodiment of the present invention.
FIG. 2 is a flowchart for explaining an additional learning step in a real-time recognition method using a plurality of machine learning models according to a preferred embodiment of the present invention.
FIG. 3 is a flowchart for explaining a stroke input step in a real-time recognition method using a plurality of machine learning models according to a preferred embodiment of the present invention.
4 is a schematic diagram showing a structure of a real-time recognition system using a plurality of machine learning models according to a preferred embodiment of the present invention.
5 is a conceptual diagram for explaining a real-time recognition method and system using a plurality of machine learning models according to a preferred embodiment of the present invention.
6A to 6D are conceptual diagrams for explaining a stroke input screen in a real-time recognition method and system using a plurality of machine learning models according to a preferred embodiment of the present invention.
FIG. 7 is a conceptual diagram for explaining the magnitude and position of a stroke on a stroke input screen in a real-time recognition method and system using a plurality of machine learning models according to a preferred embodiment of the present invention.

Before describing the present invention in detail, terms and words used herein should not be construed as being unconditionally limited in a conventional or dictionary sense, and the inventor of the present invention should not be interpreted in the best way It is to be understood that the concepts of various terms can be properly defined and used, and further, these terms and words should be interpreted in terms of meaning and concept consistent with the technical idea of the present invention.

That is, the terms used herein are used only to describe preferred embodiments of the present invention, and are not intended to specifically limit the contents of the present invention, It should be noted that this is a defined term.

Also, in this specification, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise, and it should be understood that they may include singular do.

Where an element is referred to as "comprising" another element throughout this specification, the term " comprises " does not exclude any other element, It can mean that you can do it.

Further, when it is stated that an element is "inside or connected to" another element, the element may be directly connected to or in contact with the other element, A third component or means for fixing or connecting the component to another component may be present when the component is spaced apart from the first component by a predetermined distance, It should be noted that the description of the components or means of 3 may be omitted.

On the other hand, it should be understood that there is no third component or means when an element is described as being "directly connected" or "directly connected" to another element.

Likewise, other expressions that describe the relationship between the components, such as "between" and "immediately", or "neighboring to" and "directly adjacent to" .

In this specification, terms such as "one side", "other side", "one side", "other side", "first", "second" Is used to clearly distinguish one element from another element, and it should be understood that the meaning of the element is not limited by such term.

It is also to be understood that terms related to positions such as "top", "bottom", "left", "right" in this specification are used to indicate relative positions in the drawing, Unless an absolute position is specified for these positions, it should not be understood that these position-related terms refer to absolute positions.

Furthermore, in the specification of the present invention, the terms "part", "unit", "module", "device" and the like mean a unit capable of handling one or more functions or operations, Or software, or a combination of hardware and software.

In this specification, the same reference numerals are used for the respective components of the drawings to denote the same reference numerals even though they are shown in different drawings, that is, the same reference numerals throughout the specification The symbols indicate the same components.

In the drawings attached to the present specification, the size, position, coupling relationship, and the like of each constituent element of the present invention may be partially or exaggerated or omitted or omitted for the sake of clarity of description of the present invention or for convenience of explanation May be described, and therefore the proportion or scale may not be rigorous.

Further, in the following description of the present invention, a detailed description of a configuration that is considered to be unnecessarily blurring the gist of the present invention, for example, a known technology including the prior art may be omitted.

Real-time handwriting recognition method using multiple machine learning models

First, a real-time handwriting recognition method using a plurality of machine learning models will be described with reference to FIGS.

FIG. 1 is a flowchart for explaining a main configuration of a real-time recognition method using a plurality of machine learning models according to a preferred embodiment of the present invention. As shown in FIG. 1, a real-time handwriting recognition method using a plurality of machine learning models according to an embodiment of the present invention includes a step of preparing a data set (S110), a step of generating a plurality of machine learning models S120), a step S200 of inputting an identification target of a writing style, a step S210 of applying a machine learning model, and a step S220 of outputting and storing the result.

Here, although a plurality of machine learning models are exemplarily shown as four, it is not limited thereto.

At the stage where the dataset is ready, the dataset contains a number of handwriting samples. For example, if the number "5" is entered on the terminal in handwriting (handwritten form), data for "5 " of various sizes and shapes are prepared to output a textual" 5 ". All handwriting samples related to the number "5", as well as handwriting recognition for any number, can be included in the dataset.

A plurality of different types of machine learning models may be generated by the server using the data set including the plurality of handwriting samples (S120). Wherein the server may be understood as a configuration that interacts with other system components to perform various functions and executes computer program instructions. The server may also include one or more computing devices or one or more processors.

The handwritten identification object is input on the user's terminal (S200). Preferably, the terminal is a user terminal such as a smart phone, a tablet PC, a computer equipped with a touch screen, a computer equipped with an input device capable of inputting handwriting, and the like. Here, the identification object refers to handwriting input by the user on the terminal, that is, handwriting.

The handwriting input by the terminal can be recognized as text (S210) by applying a plurality of machine learning models generated by the server. The recognized text is output as a result and stored in the server (S220).

Referring to FIG. 2, the handwriting recognition and result utilization steps will be described in detail. FIG. 2 is a flowchart for explaining an additional learning step in a real-time recognition method using a plurality of machine learning models according to a preferred embodiment of the present invention.

As shown in FIG. 2, when an identification target of a handwriting type is inputted (S200) and a machine learning model is applied (S210), a plurality of machine learning models are all applied to simultaneously inputted handwriting at the beginning to analyze the handwriting However, one of them is selected in the comparison step (S211) of the plurality of models.

For example, when four machine learning models are generated, the first to fourth models are compared, and a model that recognizes the earliest and / or most accurate handwriting as text among the models can be selected. When one machine learning model is selected while a plurality of machine learning models are applied, it is preferable that the application of the remaining machine learning models is stopped. The text result from the selected model may be displayed on the user terminal and displayed (S221).

In addition, the text result is stored again in the memory together with handwriting (S222), and the updated data set can be prepared (S310). An updated data set by the server having the memory may be utilized to generate an updated machine learning model (S320).

For example, if four machine learning models were initially created, then after the first through fourth machine learning models were used, textual results from either of them could be added to the dataset to generate an updated dataset And the first through fourth machine learning models may be generated using these updated datasets.

Next, the input step (S200) of the identification object of handwriting form will be described in more detail with reference to Fig. FIG. 3 is a flowchart for explaining a stroke input step in a real-time recognition method using a plurality of machine learning models according to a preferred embodiment of the present invention.

3, one or more strokes (for example, a unit such as a straight line, an oblique line, a curved line, or the like may be referred to as one stroke) may be input when an identification object of handwriting is input (S201) .

At this time, whether or not one input is completed can be determined when the user directly clicks the input complete button or requests an evaluation (e.g., a text result) on the input value (S202).

Accordingly, when a stroke is inputted and input from the user terminal is completed and a text value is requested, a plurality of machine learning models are applied (S210), and a text result from a machine learning model that calculates the fastest and / May be output on the user terminal (S221).

As described above, a plurality of machine learning models are applied quickly according to the input of the user, and the results can be calculated from the most suitable machine learning model among them, so real-time handwriting recognition is possible according to the present invention.

In the real-time handwriting recognition method using a plurality of machine learning models according to an embodiment of the present invention, the plurality of machine learning models preferably include at least one of a regression, a convolution neural network, a resnet, and a caption.

A computer-readable recording medium recording a program for executing a real-time handwriting recognition method using a plurality of machine learning models

A computer readable recording medium according to a preferred embodiment of the present invention is a computer readable recording medium on which a program for executing a real time handwriting recognition method using a plurality of machine learning models is recorded.

Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, and ROMs, And hardware devices specifically configured to store and execute the same program instructions. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

Real-time handwriting recognition system using multiple machine learning models

4 to 7, a real-time handwriting recognition system using a plurality of machine learning models according to the present invention will be described.

First, with reference to FIG. 4, main features of a real-time handwriting recognition system using a plurality of machine learning models according to the present invention will be described. 4 is a schematic diagram showing a structure of a real-time recognition system using a plurality of machine learning models according to a preferred embodiment of the present invention.

4, a real-time handwriting recognition system using a plurality of machine learning models according to the present invention includes a terminal 100 to which handwriting is input and a server 200 capable of communicating with the handwriting.

The terminal 100 is preferably a user terminal such as a smart phone, a tablet, a computer equipped with a touch screen, a computer equipped with an input device for inputting handwriting, and the like.

The server 100 preferably includes a translator 210, a stream unit 220, a model unit 230, and a memory 240.

 The translator 210 may be a virtual machine, a Java virtual machine, or software, which converts a programming language such as Java into byte code, interprets and executes the same. The stream unit 220 is preferably an application programming interface capable of parallel processing, in which a plurality of parallel python streams 221, 222, 223, and 224 are configurable.

The model unit 230 includes one machine learning model 231, 232, 233, and 234 for one stream 221, 222, 223, and 224, respectively. The machine learning models 231, 232, 233, and 234 thus generated may be stored in the memory 240. The memory 240 can store not only the machine learning model but also input handwriting data and output text results as data.

The process according to the exemplary handwriting input will be described with reference to Fig. 5 is a conceptual diagram for explaining a real-time recognition method and system using a plurality of machine learning models according to a preferred embodiment of the present invention.

As shown in FIG. 5, on the screen of the terminal, stroke input screens 110 and 120 may be displayed. Accordingly, the user can input a desired handwriting on the stroke input screen 110, 120.

The case where "5" 111 is input to the stroke input screen 110 and the case where the "2" 113 and the "5" 112 are input so as to be superimposed on the stroke input screen 120 are shown. At this time, as an example of the translator 210, values input through the Java virtual machine 211 are interpreted.

The interpreted value is passed to a plurality of parallel python streams 221, 222, 223, and 224, and the machine learning model to which each stream is connected is applied thereto.

5, a plurality of machine learning models may include a regression 231a, a convolution neural network (CNN) 232a, a ResNet 233a, and a CapsNet 234a. And the caps net 234a of the fourth stream 224 of the plurality of machine learning models is selected and used and the handwriting as the result 130 is recognized as text "2, 5".

At this time, the output speeds of the respective models are different, and the double caps net 234a is the fastest and the highest accuracy is obtained. As such, real-time handwriting recognition is possible through the fastest and most accurate machine learning model.

As shown in FIGS. 6A to 7, the numbers of various combinations that are heavily overlapping, and the numbers in which the various sizes and positions on the screen are inputted differently are all recognizable by the present invention.

6A to 6D are conceptual diagrams for explaining a stroke input screen in a real-time recognition method and system using a plurality of machine learning models according to a preferred embodiment of the present invention.

FIG. 7 is a conceptual diagram for explaining the magnitude and position of a stroke on a stroke input screen in a real-time recognition method and system using a plurality of machine learning models according to a preferred embodiment of the present invention.

In a preferred embodiment of the present invention, the input size and input position of the input object are not limited, and the input value may be firstly normalized to adjust the size and / or position.

Embodiments of the present invention include a computer-readable medium having program instructions for performing various computer-implemented operations. The computer-readable medium may include program instructions, local data files, local data structures, etc., alone or in combination. The media may be those specially designed and constructed for the present invention or may be those known to those skilled in the computer software.

Further, those skilled in the art will appreciate that a computing device may be implemented in firmware (e.g., an application-specific integrated circuit), hardware, or a combination of software, firmware, and hardware. Those skilled in the art will also appreciate that the functions of the various computing devices may be combined or integrated into a single computing device or may be distributed to one or more other computing devices without departing from the scope of the exemplary embodiments of the present invention. It should be understood. The server may be a software module and may also be referred to simply as a module.

As described above, according to the present invention, when an arbitrary image is input, recognition of a specific object in the image can be performed quickly and accurately.

100: terminal
110, 120: Stroke input screen
111: first stroke
112: second stroke
113: Third stroke
130: Results
200: Server
210: Translator
211: The Java Virtual Machine
220:
221: first stream
222: second stream
223: Third stream
224: fourth stream
230: Model part
231, 231a: First model
232, 232a: second model
233, 233a: the third model
234, 234a: fourth model
240: Memory

Claims (11)

(a) a plurality of different types of machine learning models are generated by a server using a data set including a plurality of handwriting samples;
(b) inputting a handwriting including characters superimposed on the terminal;
(c) applying the plurality of machine learning models;
(d) selecting any one of the plurality of machine learning models; And
(e) using the selected one machine learning model to recognize the superimposed text as separate texts instead of superimposed text,
Wherein the server further comprises a translator and a stream portion including a plurality of streams connected to the translator,
The overlapped character is interpreted as the value input through the translator,
Wherein the analyzed value is transmitted to the stream unit, and a machine learning model connected to each stream is applied, and the overlapped characters are recognized as separate states,
A real - time handwriting recognition method using multiple machine learning models.
The method according to claim 1,
Wherein the one machine learning model selected in the step (d) is one in which the result output speed is the fastest among the plurality of machine learning models,
A real - time handwriting recognition method using multiple machine learning models.
The method according to claim 1,
In the step (d), the one selected machine learning model may be one in which the accuracy of the result among the plurality of machine learning models is the highest,
A real - time handwriting recognition method using multiple machine learning models.
The method according to claim 1,
Wherein when the one machine learning model is selected in the step (d) while the plurality of machine learning models are applied in the step (c), the application of the remaining machine learning model is stopped,
A real - time handwriting recognition method using multiple machine learning models.
The method according to claim 1,
Wherein the plurality of machine learning models comprises at least one of Regression, Convolutional Neural Network (CNN), ResNet, and CapsNet.
A real - time handwriting recognition method using multiple machine learning models.
The method according to claim 1,
(f) further storing and updating the handwriting and the text in the data set; And
(g) by the server, the updated data set is utilized to generate a plurality of different types of machine learning models,
A real - time handwriting recognition method using multiple machine learning models.
A computer-readable recording medium recording a program for executing a real time handwriting recognition method using a plurality of machine learning models according to any one of claims 1 to 6.
A terminal for inputting a handwriting including overlapping characters; And
And a server capable of communicating with the terminal,
The server comprises:
A plurality of machine learning models, and
A memory for storing a data set and the plurality of machine learning models,
One of the plurality of machine learning models is selected and used, and the overlapped text is recognized as a separate text instead of the overlapped text,
Wherein the server further comprises a translator and a stream portion including a plurality of streams connected to the translator,
The overlapped character is interpreted as the value input through the translator,
Wherein the analyzed value is transmitted to the stream unit, and a machine learning model connected to each stream is applied, and the overlapped characters are recognized as separate states,
A real - time handwriting recognition system using multiple machine learning models.
delete 9. The method of claim 8,
The translator is a Java virtual machine,
A real - time handwriting recognition system using multiple machine learning models.
9. The method of claim 8,
Wherein the plurality of machine learning models comprise at least one of regression, convolutional neural network, resnet, and < RTI ID = 0.0 >
A real - time handwriting recognition system using multiple machine learning models.

Images