KR20240006915A - Method for Decomposition of Phonemes Based on Number of Strokes - Google Patents

Abstract

One purpose of the present invention is to effectively perform phoneme decomposition of Hangul handwriting data and at the same time determine whether each phoneme has been entered in the correct order.
To this end, the stroke number-based phoneme decomposition method according to an embodiment of the present invention includes the step of determining the number of strokes for each phoneme of the corresponding Hangul character for Hangul handwriting data input by the learner - the Hangul handwriting data is input by the learner. Contains time-series two-dimensional coordinate-based data - determining, based on the number of strokes for each phoneme, whether any input of the Hangul handwriting data corresponds to one of the initial consonants, middle consonants, and final consonants, and the determination A step of decomposing phonemes of the Hangul handwriting data based on the number of strokes for each phoneme of the Hangul handwriting data and the location of the determined phoneme of the Hangul handwriting data to determine whether the learner inputs Hangul letters based on a determined stroke order. It includes a step of determining whether or not.

Description

Method for Decomposition of Phonemes Based on Number of Strokes}

The present invention relates to a method for decomposing phonemes based on the number of strokes, and more specifically, to a method for decomposing phonemes in Hangul handwriting data based on the number of strokes of corresponding Hangul letters in the Hangul handwriting data.

As enthusiasm for education in various languages such as Korean, English, Japanese, and Chinese heats up, parents recently began teaching children in various languages, such as Korean and English, from as early as age 4. In particular, in the case of Hangul, which is the core of early education, according to statistics from the Korea Institute for Curriculum and Evaluation, the rate of Hangul education before entering school reached 91.6%.

The existing method of teaching Korean using materials such as Korean workbooks during the care time of parents or caregivers became the mainstream, and was gradually transitioning to teaching online or offline through applications such as smartphones or tablets. . However, these Hangul education methods were either a low-cost, one-way Hangul education method based on a preset curriculum that was not related to the child's Hangul level, or an expensive, two-way Hangul education method that included guidance from a workbook teacher.

For this reason, when it was necessary to provide children's Hangul education at a low cost, even if Hangul education was conducted using an application, it was difficult to arouse children's interest because the level of difficulty was too high or too low. Alternatively, there was a problem in which caregivers had to devote a lot of teaching time to children's Korean language education. Additionally, from the perspective of parents and caregivers, there was a very difficult problem of having to choose which one was most appropriate among the plethora of options, including more than 4,000 Hangul education worksheets and numerous Hangul education applications available on the market.

In addition, even if an expensive two-way Hangul education method was used in which a workbook teacher visits and provides guidance, the workbook teacher usually only provides 15 minutes of education per week, and there was a problem in that parents had to take responsibility for Hangul education themselves for the remaining time.

One purpose of the present invention is to effectively perform phoneme decomposition of Hangul handwriting data and at the same time determine whether each phoneme has been entered in the correct order.

However, the problem to be solved by the present invention is not limited to the above-mentioned, and includes purposes that are not mentioned but can be clearly understood by those skilled in the art from the description below. can do.

Hereinafter, specific means for achieving the purpose of the present invention will be described.

The stroke number-based phoneme decomposition method according to an embodiment of the present invention includes the step of determining the number of strokes for each phoneme of the corresponding Hangul character for Hangul handwriting data input by the learner - the Hangul handwriting data is a time-series 2 input by the learner. Containing dimensional coordinate-based data - determining, based on the number of strokes for each phoneme, whether any input of the Hangul handwriting data corresponds to any one of initial consonants, middle consonants, and final consonants; and based on the determination, Decomposing phonemes of the Hangul handwriting data, and determining whether the learner inputs Hangul letters based on a determined stroke order based on the number of strokes for each phoneme of the Hangul handwriting data and the determined location of the phoneme in the Hangul handwriting data. It includes steps to:

As described above, according to the present invention, the following effects are achieved.

According to one embodiment of the present invention, not only can phoneme decomposition of Hangul handwriting data be efficiently performed just by utilizing low computational resources, but it is also determined whether each phoneme is input in the correct order, so that if the phoneme input order is incorrect, it is possible to efficiently decompose Korean handwriting data. Since the multifaceted process can be terminated early, the overall efficiency of the stroke recognition device can be improved.

However, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The following drawings attached to this specification illustrate preferred embodiments of the present invention and serve to further understand the technical idea of the present invention along with the detailed description of the invention, so the present invention is limited only to the matters described in such drawings. It should not be interpreted as such.
Figure 1 is a schematic diagram showing an artificial intelligence-based stroke order recognition device for a young learner's Korean handwriting according to an embodiment of the present invention.
Figure 2 is an exemplary diagram of an artificial intelligence-based handwriting recognition module according to an embodiment of the present invention.
Figure 3 is an exemplary diagram for explaining the operating principle of the stroke number-based phoneme decomposition module according to an embodiment of the present invention.
Figure 4 is an exemplary diagram for explaining the operating principle of an artificial intelligence-based phoneme decomposition module according to another embodiment of the present invention.
Figure 5 is a diagram explaining the operating principle of an artificial intelligence-based stroke order identification module according to an embodiment of the present invention.
Figure 6 is a diagram illustrating preprocessing performed on phoneme information according to an embodiment of the present invention.
Figure 7 is a diagram schematically explaining a pre-learning session of the artificial neural network of the stroke order identification module according to an embodiment of the present invention.
Figure 8 is a flowchart of an artificial intelligence-based stroke order recognition method for an infant learner's Korean handwriting according to an embodiment of the present invention.

Hereinafter, with reference to the attached drawings, an embodiment by which a person skilled in the art can easily carry out the present invention will be described in detail. However, the present invention may be implemented in many different forms and is not limited to the examples described herein. Additionally, in explaining in detail the operating principle of a preferred embodiment of the present invention, if a detailed description of a related known function or configuration is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

In addition, the same reference numerals are used for parts that perform similar functions and actions throughout the drawings. Throughout the specification, when a specific part is said to be connected to or above/below another part, this means not only directly connected or above/below, but also indirectly through other components in between. Also includes cases where it is connected or above/below.

Furthermore, when it is said that a part "includes" a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Terms such as “first” and “second” may be used to describe various components, but these components should not be limited by these terms. That is, terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term “and/or” includes any of a plurality of related stated items or a combination of a plurality of related stated items.

Additionally, unless specifically stated otherwise, the expression “singular” is used herein to include one or more entities. Finally, as used herein, “or” means “or” non-exclusively, unless specifically stated otherwise.

In addition, the terms used in the present invention are general terms that are currently widely used as much as possible while considering the functions in the present invention, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

In the description of the invention below, Hangul is a single sound (phoneme) character composed of consonants (taeksho) and vowels (singles), and syllables (sounds) are composed of an initial sound (initial sound), a middle sound (medial sound), and a final sound (final sound). It consists of three sets of single sounds (phonemes), with consonant sounds (consonants) used for the first sound (initial consonant) and final sounds (final consonants), and single sounds (vowels) used for the middle sound (neutral voice).

In modern Hangul, there are 14 consonants that make single sounds, including ㄱ, ㄴ, ㄷ, ㄹ, and ㅁ, and single sounds (vowels) include 10 characters, including ㅏ, ㅑ, ㅓ, and ㅕ. There are five double consonants (double consonants) that make gospel sounds, including ㄲ, ㄸ, ㅃ, ㅆ, and ㅉ, and double single sounds (double vowels) include 11 letters, including ㅐ, ㅒ, ㅔ, and ㅖ. In addition, in modern Hangul, the consonants used when there is a final sound (jongson) are single consonants or side consonants. All consonants (consonants) are used in single consonants, and there are 13 consonants including ㄲ, ㅆ, ㄳ, and ㄵ.

Modern Hangul is able to write 11,172 letters (19 initial sounds × 21 middle sounds × (27 final sounds + 1 no final sound)) by stringing together letters. Of the 11,172 characters, 399 are non-consonant letters and 10,773 are consonant letters.

Due to grammar regulations, there are fewer syllables actually used in modern Korean standard language. The sounds of Korean are made up of the first sound + the middle sound (+ the final sound). In standard language, all 19 consonants are used for the first sound, but the ㅇ placed at the first sound is not made. The final sounds are grouped into 7 types according to the hexaphonological law, and there are a total of 8, including those without final sounds, so the pronunciation of modern Korean is 19 initial sounds × 21 middle sounds × 8 final sounds = 3,192 sounds. According to the standard pronunciation method, the double vowels ㅑ, ㅒ, ㅕ, ㅖ, ㅛ, ㅠ after the palatal sounds ㅈ, ㅉ, ㅊ are sounded as the single vowels ㅏ, ㅐ, ㅓ, ㅔ, ㅗ, ㅜ, so the initial sound is 3 × the middle sound is 6 × the final sound. 8 = 144 sounds are omitted, and the middle sound [ㅢ] with the initial sound (following the initial sound, not ㅇ) is excluded, except for ㄴ (in the case of ㄴ, it is recognized as a different phoneme according to palatalization) [ㅣ] (Korean orthography Article 9 and Standard Pronunciation Article 5 Proviso 3) 17 initial sounds × 1 middle sound × 8 final sounds = 136 sounds are missing again. Therefore, the number of sound nodes actually used in the current standard Korean language is 3192 - 144 - 136 = 2,912.

Overview of artificial intelligence-based stroke order recognition device

Figure 1 is a schematic diagram showing an artificial intelligence-based stroke order recognition device for a young learner's Korean handwriting according to an embodiment of the present invention. Referring to Figure 1, the artificial intelligence-based stroke order recognition device (l00) for the Korean language handwriting of an infant learner according to an embodiment of the present invention includes a handwriting recognition module (1), a phoneme decomposition module (2), and a stroke order identification. It may include module (3).

According to one embodiment of the present invention, the handwriting recognition module 1 of the artificial intelligence-based stroke order recognition device 100 for the Korean language handwriting of an infant learner is configured to recognize the input by the learner from the application module 5 of the learner client 4. You can receive Hangul handwriting data. The learner's Hangul handwriting data is time-series two-dimensional coordinate-based handwriting data. For example, the Hangul handwriting data may be data that records the two-dimensional coordinates of the learner's touch input in time order. Such Hangul handwriting data may be input by touching the learner's finger or stylus through the touch screen of the learning client 4, or through other methods such as a mouse, and there is no limitation on the input method.

According to one embodiment, the handwriting recognition module 1 can recognize which letter the Korean handwriting data corresponds to through deep learning using an artificial neural network module. The artificial neural network module used in the handwriting recognition module 1 may be, for example, a CNN (Convolution Neural Network)-based artificial neural network using a GoogLenet neural network, a Lenet neural network, or the like. The specific configuration of the handwriting recognition module 1 will be described in detail later.

According to one embodiment, the phoneme decomposition module 2 uses deep learning using an artificial neural network module to identify each phoneme of the letter represented by the Korean handwriting data recognized through the handwriting recognition module 1, for example, initial consonant, Neutral and final consonants can be decomposed. The artificial neural network module included in the phoneme decomposition module 2 may be, for example, YOLO, DPM, R-CNN, etc., which will be described in detail later.

In addition, according to another embodiment, the phoneme decomposition module 2 determines what input in the Hangul handwriting data is based on the number of strokes, based on what letter the Hangul handwriting data recognized through the handwriting recognition module 1 represents. Phonemes can also be decomposed by determining whether they are initial, medial, or final consonants. For example, when writing the letter “Ram,” “ㄹ” is written with 3 strokes, “ㅐ” with 3 strokes, and “ㅁ” with 3 strokes, so it is written with a total of 9 strokes, so the Korean writing “Ram” The phoneme can be decomposed by considering that the first three strokes correspond to the initial consonant, the next three strokes correspond to the middle consonant, and the last three strokes correspond to the final consonant.

According to one embodiment, the stroke order identification module 3 inputs a time-series, two-dimensional coordinate-based cursive database corresponding to the decomposed phoneme into the artificial neural network module, and determines whether each stroke of the corresponding phoneme is input in a certain order. Recognition information can be output. Here, the two-dimensional coordinate cursive database of phonemes may include information about the form of the input Hangul handwriting data as well as information sequentially expressing the coordinates. The artificial neural network module used in the stroke order identification module 3 may be, for example, Long Short Term Memory (LSTM), Recurrent Neural Network (RNN), etc., which will be described in detail later.

Additionally, according to one embodiment, the stroke order identification module 3 may transmit feedback to the application module 5 of the learner client 4 based on the output stroke order recognition information. This feedback may be intended to inform the learner of information about the incorrect stroke order (for example, which stroke number of the Korean handwriting entered by the learner was incorrect), and is not limited to specific examples of the information.

The artificial intelligence-based stroke order recognition device 100 for an early childhood learner's Hangul handwriting according to an embodiment of the present invention is a specific web server, a virtual server such as a cloud server, a computing device such as a smartphone, a tablet PC, or a desktop PC. It may be configured to be processed by a processing module and stored in a memory module of each computing device. However, this device 100 may be distributedly processed by the processing modules of two or more computing devices and distributedly stored in the memory modules of two or more computing devices, and this is due to the fact that the artificial intelligence-based The stroke order recognition device 100 may be designed in various ways depending on the specific implementation method.

According to one embodiment of the present invention, it is possible to not only determine whether the early childhood learner wrote the Hangul handwriting in an appropriate form, but also determine whether the stroke order of the Hangul handwriting was entered in the correct order, thereby improving the writing education of the early childhood learner more systematically. There are effects that can be performed.

However, in this specification, the explanation is focused on the learner's Hangul handwriting data, that is, Hangul input, but it is common knowledge in the field of technology that the same or similar principles can be applied to other languages such as English, Chinese, Japanese, French, etc. Anyone equipped with this will be able to understand it easily.

Artificial intelligence-based handwriting recognition module

Figure 2 is an exemplary diagram of an artificial intelligence-based handwriting recognition module according to an embodiment of the present invention.

Referring to Figure 2, the artificial intelligence-based handwriting recognition module 1 can receive the learner's Hangul handwriting data as input, recognize and output Hangul letters corresponding to the Hangul handwriting data. In the example of Figure 2, a two-dimensional image (i.e., Korean handwriting data) containing the Korean character "Nat" is input to the handwriting recognition module (1), and the Korean character "Nat" is input as an output to the handwriting recognition module (1). ) is recognized and printed.

The structure and operating principle of the artificial intelligence-based handwriting recognition module 1 will be described with reference to FIG. 2. The handwriting recognition module 1 may include a normalization module, an encoding module, a plurality of inception modules, pooling, and a fully connected (FC) layer.

First, the grid size of the early childhood learner's Hangul handwriting data (for example, a two-dimensional image of Hangul handwriting) can be normalized through a normalization module. For example, the normalization module can convert Hangul handwriting data with different grid sizes into data with a grid size of 60×60, and for this purpose, image preprocessing techniques such as linear interpolation can be used.

Normalized Hangul handwriting data can be input to an encoding module that includes a convolution layer (convolution), a pooling layer (max_pool), an activation layer (ReLU), and a BN (batch_normalization) layer. And by performing feature extraction from these layers, the encoded Hangul handwriting data can be output from the encoding module.

Next, the encoded Hangul handwriting data can be input into a plurality of inception modules. At this time, a plurality of inception modules basically maintain the same channel, and among them, at least one inception module may be an improved inception module that reduces the amount of computation by factorizing the basic inception module. According to one example, 512 channels in a 7×7 grid can be created through the operation of the inception module, and the 7×7 data is averaged and then divided into 1×1×512 nodes for classification through the FC layer. It can be transformed.

Finally, classification of Hangul handwriting data can be performed through the FC layer to determine which Hangul character the Hangul handwriting data entered by the infant learner corresponds to.

When using MSE (Mean Square Error) in the pre-training session of this artificial intelligence-based handwriting recognition module (1), the error function is It can be , where X is the predicted value through the model and Y is the actual correct answer. MSE uses the L2 distance between the predicted value and the actual value as the error value.

As another example, CE (Cross Entropy) is used. You can also use the error function of Here, X is the predicted value through the model, Y is the actual answer, and a very small value called ε can be added to prevent log(

However, the handwriting recognition module 1 as described above is only an example, and various artificial neural network modules or other modules may be used, but there is no limitation thereto.

Stroke order-based phoneme decomposition module

Figure 3 is an exemplary diagram for explaining the operating principle of the stroke number-based phoneme decomposition module according to an embodiment of the present invention.

Referring to FIG. 3, according to one embodiment, the phoneme decomposition module 2 determines the number of strokes based on the Hangul handwriting data input by the infant learner and the corresponding Hangul characters of the Hangul handwriting data determined through the handwriting recognition module 1. Based on this, phoneme information can be generated and output by decomposing phonemes by determining whether any input in the Hangul handwriting data is an initial, medial, or final consonant.

For example, when writing the letter “Ram,” “ㄹ” is written with 3 strokes, “ㅐ” with 3 strokes, and “ㅁ” with 3 strokes, so it is written with a total of 9 strokes, so the Korean writing “Ram” Considering that the first 3 strokes correspond to the initial consonant, the next 3 strokes correspond to the middle consonant, and the last 3 strokes correspond to the final consonant, the phoneme is decomposed into phoneme information corresponding to (initial consonant, middle consonant) or (initial consonant, middle consonant, and final consonant). can be output.

As a non-limiting example, when the phoneme decomposition module 2 decomposes the phonemes of a letter, the coordinates of the handwriting data of each phoneme classified based on the stroke order among the Hangul handwriting data are located at the top, left, right, and bottom. Bounding boxes 41, 42, and 43 for each phoneme are created based on the coordinates (or coordinates with a predetermined pixel margin more than the coordinates at the top, left, right, and bottom), and the data within each bounding box is generated. can be output as phoneme information.

Meanwhile, the phoneme decomposition mechanism of the stroke number-based phoneme decomposition module (2) is premised on the fact that the early childhood learner has accurately entered Hangul letters based on the designated stroke order. However, for example, in the case where the number of strokes in the Hangul handwriting data for Hangul letters is different from the designated number of strokes (for example, the number of strokes for the letter “Ram” was entered as 10 instead of 9), or when an infant learner uses the Korean handwriting data This may occur when each phoneme is entered in a different order than the initial, middle, and final consonants. In this way, if the early childhood learner does not input Hangul letters based on the designated phoneme input order, the stroke number-based phoneme decomposition module may fail to decompose the phonemes of Hangul handwriting data or an error may occur.

Accordingly, the stroke number-based phoneme decomposition module 2 may go through a process of identifying whether the input order of each phoneme conforms to the specified order. Specifically, the initial consonant of Hangul letters is usually located at the top or middle of the left, the middle consonant is located at the middle or right of the horizontal length, or the middle or bottom of the vertical length, and the final consonant is usually located at the bottom. Therefore, with this in mind, the stroke-based phoneme decomposition module (2) is a process of identifying cases where data is input at positions that do not correspond to the designated positions of the initial, middle, and final consonants through time series data of the coordinates of Hangul handwriting data. can proceed additionally. As a non-limiting example, coordinate data of a specific phoneme of Hangul handwriting data is received, the degree to which the received coordinate data matches the coordinate data of a given location of the phoneme is quantified, and the quantified degree of match is less than a threshold. In this case, it may be determined that the phoneme in question was not entered in a given order. Through this process, the stroke number-based phoneme decomposition module 2 can determine whether each phoneme of the Korean handwriting data has been entered in a given order.

And, as a result of the determination, if phoneme decomposition based on stroke number fails or an error occurs because each phoneme of the Korean handwriting data is not entered in the specified order, the stroke number-based phoneme decomposition module (2) is used in the application module (5) of the learner application (4). ), a message notifying that the stroke order of the Hangul handwriting data entered by the learner is incorrect can be output. Additionally, in this case, the process may be terminated without phoneme information being transmitted to the stroke order identification module 3.

Through this stroke-based phoneme decomposition module, not only can phoneme decomposition of Hangul handwriting data be performed efficiently just by utilizing low computational resources, but it is also possible to determine whether each phoneme has been entered in the correct order, so if the phoneme input order is incorrect, Since the multifaceted process can be terminated early, the overall efficiency of the stroke recognition device can be improved.

Artificial intelligence-based phoneme decomposition module

Figure 4 is an exemplary diagram for explaining the operating principle of an artificial intelligence-based phoneme decomposition module according to another embodiment of the present invention.

Referring to FIG. 4, according to one embodiment, the artificial intelligence-based phoneme decomposition module 2 uses deep learning using an artificial neural network module to identify Hangul characters represented by Hangul handwriting data recognized through the handwriting recognition module 1. It can act to decompose each phoneme, for example, initial consonant, middle consonant, and final consonant.

The artificial neural network module included in this artificial intelligence-based phoneme decomposition module 2 may be, for example, at least one of YOLO, DPM, and R-CNN. According to a non-limiting example, the artificial intelligence-based phoneme decomposition module 2 may use YOLO as an artificial neural network module, where YOLO's object detection or image segmentation system is different from R-CNN. Since it passes through the convolution neural network only once, it has the advantage of being fast and enabling real-time object detection.

This artificial intelligence-based phoneme decomposition module (2) can receive character-recognized Hangul handwriting data as input and output phoneme information as its output. The output phoneme information is data within the bounding boxes 41, 42, and 43 of each phoneme, and may include time-series two-dimensional coordinate-based data according to the input order of the learner's Hangul handwriting data. In addition, optionally, the phoneme information may include data on which letter the corresponding phoneme is (e.g., which letter of the Korean consonant or vowel) and/or phoneme input order data indicating the input order of each phoneme. there is.

According to one embodiment, the artificial intelligence-based phoneme decomposition module 2 is based on the phoneme input order data of each phoneme of the Hangul handwriting data input by the learner, and the input order of each phoneme is a predetermined order of the phonemes of the corresponding Hangul letter ( For example, in the case of "RAM" illustrated in FIG. 4, if it is determined that the input is different from the order (41 -> 42 -> 43), the phoneme decomposition module (2) is the application module (5) of the learner application (4) A message notifying that the stroke order of the Hangul handwriting data entered by the learner is incorrect can be displayed. In this case, the artificial intelligence-based phoneme decomposition module 2 can terminate the process as is without transmitting phoneme information to the stroke order identification module 3.

Through this artificial intelligence-based phoneme decomposition module, not only can high-accuracy phoneme decomposition of Hangul handwriting data be performed, but it can also be determined whether each phoneme has been entered in the correct order. If the phoneme input order is incorrect, the process can be started early. Since it can be terminated at , the effect of improving the overall efficiency of the stroke recognition device occurs.

Artificial intelligence-based stroke order identification module

Figure 5 is a diagram explaining the operating principle of an artificial intelligence-based stroke order identification module according to an embodiment of the present invention.

The artificial intelligence-based stroke order identification module (3) in Figure 5 receives phoneme information of Hangul handwriting data from the phoneme decomposition module (2) in an inference session using a pre-trained artificial neural network module, and processes each phoneme information. Stroke order recognition information can be output.

To explain the operation process of the artificial intelligence-based stroke order identification module 3, first, the stroke order identification module 3 can perform preprocessing on each received phoneme information, and this preprocessing involves upsampling or downsampling. May include downsampling.

In addition, the preprocessed phoneme information may be input to an artificial neural network module, for example, an LSTM network and a Fully Connected (FC) layer (or Dense layer), and finally stroke order recognition information may be output. However, the LSTM module and FC layer here are only examples, and RNN modules, CNN modules, etc. may be used, but are not limited thereto.

Additionally, the stroke order identification module determines whether the stroke order of the stroke order recognition information output from the artificial neural network module is input in a given order, that is, performs stroke order correctness determination.

And feedback can be sent to the learner client (4) according to the result of the stroke order correctness determination. This feedback may be intended to provide learners with information about incorrect stroke order, and is not limited to the specific content of the feedback. For example, the stroke order identification module 2 can output a message to the application module 5 of the learner application 4 that informs which stroke order of the Korean handwriting data entered by the learner is incorrect.

Figure 6 is a diagram illustrating preprocessing performed on phoneme information according to an embodiment of the present invention.

Preprocessing of phonemic information is intended to improve the recognition rate of Korean consonant and vowel cursive, and to deal with the problem that the length of input data varies depending on the type of character or the input interface of the device. As preprocessing for phoneme information, at least one of upsampling/downsampling, zero-padding, and windowing techniques can be used. Figure 6 is a non-limiting example in which upsampling/downsampling is performed as preprocessing for phoneme information. As in the example shown above in Figure 6, the number of data for phoneme "ㄱ" is If 25 pixels are less than the standard value (30), pixel data can be added to bring the number of data to 30 through upsampling. In addition, as in the example shown below in FIG. 6, if the number of data for the phoneme “ㅂ” is 40, which is more than the standard value, some pixel data can be removed to bring the number of data to 30. However, the specific method of preprocessing phoneme information, including upsampling/downsampling, may be modified in various ways and is not limited thereto.

Figure 7 is a diagram schematically explaining a pre-learning session of the artificial neural network of the stroke order identification module 3 according to an embodiment of the present invention.

Referring to FIG. 7, in the dictionary learning session of the artificial neural network of the stroke order identification module 3, first, a plurality of phoneme information for dictionary learning may be input to the artificial neural network. Such plural phoneme information for prior learning can be obtained from a Korean cursive database previously collected from multiple learners, and the Korean cursive database includes a time-series two-dimensional coordinate-based database according to the input order of the Korean handwriting data of multiple learners. can do. By processing a plurality of phoneme information for dictionary learning through a combination of an LSTM module and an FC layer suitable for time-series processing of data, the artificial neural network of the stroke order identification module 3 can output stroke order recognition information.

The stroke order recognition information (i.e., recognition information about the order in which phoneme strokes are input) output by the artificial neural network module of the stroke order identification module 3 in the dictionary learning session is information about the order in which the strokes of phonemes for dictionary learning were actually input. It can be compared with the actual stroke order information. In addition, the artificial neural network module may be pre-trained so that the error (-∑ y _i log p _i ) between the stroke order recognition information and the actual stroke order information is minimized. For example, the artificial neural network module may adopt a method of updating the weight of the hidden layer based on the error between stroke order recognition information and actual stroke order information. A large number of phoneme information for pre-learning is input to each node of the input layer such as x1, x2, x3 of the LSTM module, and based on the weight such as w1, passes through hidden layers such as h1, h2, and h3 and is based on a cost function such as softmax. The judgment prediction information predicted by is output to the output layer called y1. And the weight of the artificial neural network module can be updated through back propagation based on the error (-∑ y _i log p _i ) between the stroke order recognition information and the actual stroke order information. In this way, the inference session of the stroke order identification module 3 of FIG. 5 can be implemented using the artificial neural network of the pre-trained stroke order identification module 3.

Artificial intelligence-based stroke order recognition method for early childhood learners’ Hangul handwriting

Figure 8 is a flowchart of an artificial intelligence-based stroke order recognition method for an infant learner's Korean handwriting according to an embodiment of the present invention.

Referring to FIG. 8, first, in step S810, characters of the learner's Hangul handwriting data received from the learning client can be recognized. The learner's Hangul handwriting data may be time-series two-dimensional coordinate-based data, and may be input by touching a finger or stylus through the touch screen of the learning client, or through other methods such as a mouse.

In step S820, phoneme information can be generated by decomposing the phonemes of the recognized Korean handwriting data. As described above, this decomposition of phonemes may be based on stroke numbers or may be based on artificial intelligence. Additionally, the phoneme information generated at this time may include phoneme input order data indicating the input order of each phoneme in the Hangul handwriting data entered by the learner.

In step S830, it can be determined whether each phoneme of the Hangul handwriting data has been input in a predetermined order based on phoneme information (or phoneme input order data). If it is determined that each phoneme has not been entered in the prescribed order, a message notifying that the stroke order of the Korean handwriting data entered by the learner is incorrect can be output to the learner's client. Also, in this case, the process may be terminated without moving to the next step.

In step S840, phoneme information can be input into the artificial neural network module to output stroke order recognition information, which is the order in which each stroke of the phoneme is input by the learner. Additionally, in step S840, preprocessing may be performed on the phoneme information before inputting the phoneme information to the artificial neural network module, and this preprocessing may include upsampling or downsampling. In addition, the preprocessed phoneme information may be input to an artificial neural network module, for example, an LSTM network and a Fully Connected (FC) layer (or Dense layer), and finally stroke order recognition information may be output. Additionally, the stroke order identification module may determine whether the stroke order of the stroke order recognition information for each phoneme output from the artificial neural network module is input in a predetermined order, that is, perform stroke order error determination.

In step S850, feedback may be transmitted to the learner client based on the stroke order recognition information. In other words, if it is determined that the stroke order of the Korean handwriting data entered by the learner is incorrect as a result of performing stroke order error determination based on stroke order recognition information, feedback on this can be provided to the learner.

According to one embodiment of the present invention, not only can it be determined whether the early childhood learner has written the Hangul handwriting in an appropriate form, but it is also possible to determine whether the stroke order of the Hangul handwriting has been entered in the correct order, thereby improving the writing education of the early childhood learner. It has an effect that can be carried out systematically.

computer-readable recording medium

It is obvious that each step or operation according to the embodiments of the present invention can be performed by a computer including one or more processors by executing a computer program stored in a computer-readable recording medium.

Each instruction stored in the above-described recording medium can be implemented through a computer program programmed to perform each corresponding step, and such computer program may be stored in a computer-readable recording medium and can be executed by a processor. do. A computer-readable recording medium may be a non-transitory readable medium. At this time, a non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as registers, caches, and memories. Specifically, programs for performing the various methods described above include semiconductor memory devices such as erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), and flash memory devices, internal hard disks, and removable memory devices. It may be provided and stored in non-transitory readable media such as magnetic disks such as disks, optical-magnetic disks, and non-volatile memory including CD-ROM and DVD-ROM disks.

Methods according to various examples disclosed in this document may be provided and included in a computer program product. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or online through an application store (e.g. Play Store ^™ ). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or created temporarily in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

As described above, a person skilled in the art to which the present invention pertains will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. Therefore, the above-described embodiments should be understood in all respects as illustrative and not restrictive. The scope of the present invention is indicated by the claims described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and the equivalent concept should be construed as being included in the scope of the present invention.

The features and advantages described herein are not all-inclusive, and many additional features and advantages will become apparent to those skilled in the art upon consideration of the drawings, specification, and claims. Moreover, it should be noted that the language used herein has been selected primarily for readability and teaching purposes and may not be selected to describe or limit the subject matter of the invention.

The foregoing description of embodiments of the invention has been presented for purposes of illustration. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will appreciate that many modifications and variations are possible in light of the above disclosure.

Therefore, the scope of the present invention is not limited by the detailed description, but by any claims of the application based thereon. Accordingly, the disclosure of embodiments of the invention is illustrative and does not limit the scope of the invention as set forth in the claims below.

100: Artificial intelligence-based stroke order recognition device for early childhood learners’ Hangul handwriting
1: Handwriting recognition module
2: Phoneme decomposition module
3: Stroke order identification module
4: Learner Client
5: Application module

Claims (1)

As a stroke number-based phoneme decomposition method,
A step of determining the number of strokes for each phoneme of a Hangul character corresponding to the Hangul handwriting data input by the learner - the Hangul handwriting data includes time-series two-dimensional coordinate-based data input by the learner - and
Based on the number of strokes for each phoneme, determining whether any input of the Hangul handwriting data corresponds to any one of the initial consonant, middle consonant, and final consonant;
Decomposing phonemes of the Hangul handwriting data based on the determination;
A step of determining whether the learner inputs Hangul letters based on a determined stroke order based on the number of strokes for each phoneme of the Hangul handwriting data and the determined location of the phoneme in the Hangul handwriting data.
A stroke number-based phoneme decomposition method including.