KR102443304B1 - Device for Recognition and Recognition Method of Input Language Through Machine Learning - Google Patents

Abstract

The present invention recognizes a type of input language by sensing a user's movement, and more particularly, relates to an apparatus and method for recognizing an input language through machine learning. To this end, an inertial sensor 130 attached to a body part of the user 5 to detect a motion occurring when the user 5 inputs a language; a machine learning unit 330 for machine learning an operation related to language input; an inference unit 230 for inferring a language as a first language based on a learning model from the machine learning unit 330 and an operation signal of the inertial sensor 130; a second control unit 210 provided in an input device into which a language is input and determining the input language as a second language; a second control unit 210 that determines whether the first language and the second language are the same language; and warning means for generating a warning when the first language and the second language are different from each other.

Description

Device for Recognition and Recognition Method of Input Language Through Machine Learning

The present invention recognizes a type of input language by sensing a user's movement, and more particularly, relates to an apparatus and method for recognizing an input language through machine learning.

In many cases, the language (eg Korean) is input through the keyboard on the computer software (eg word processor, chatting, Excel, PowerPoint, etc.). However, even when the user inputs Korean, there are cases in which the alphabet is input because the keyboard of the computer is converted into English by mistake or automatically. Or, conversely, even if the user wants to input English, the keyboard of the computer is converted into Korean, so that Hangul is inputted.

When such an error input occurred, the user had to erase all the entered characters, switch the keyboard language to the desired Korean, and then re-enter Korean. Some word processors recognize these error inputs and automatically convert them into Hangul, but they are limited to specific software and have not been generally used.

1. Republic of Korea Patent Registration No. 10-1809278 (Device and method for inputting text by detecting user's movement), 2. Korean Patent Laid-Open No. 10-2006-0030456 (a method for inputting characters and functions combining user's key input and movement and a terminal using the same).

Accordingly, the present invention has been devised to solve the above problems, and the object of the present invention is to recognize an input operation in a wearable device attached to a user, and input the first language inferred according to the input operation. An object of the present invention is to provide an apparatus for recognizing an input language and a method for recognizing an input language through machine learning to compare and determine whether it is the same as a second language.

A second object of the present invention is to provide an apparatus for recognizing an input language and a method for recognizing an input language through machine learning that can accurately infer a first language input as an input operation by machine learning the characteristics of an input operation for each language.

A third object of the present invention is to provide an apparatus and method for recognizing an input language through machine learning that can warn a user by determining whether an inferred first language and an input second language are the same.

However, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned are clearly to those of ordinary skill in the art to which the present invention belongs from the description below. can be understood

In order to achieve the above technical task, an inertial sensor 130 attached to a body part of the user 5 to detect a motion occurring when the user 5 inputs a language; a machine learning unit 330 for machine learning an operation related to language input; an inference unit 230 for inferring a language as a first language based on a learning model from the machine learning unit 330 and an operation signal of the inertial sensor 130; a second control unit 210 provided in an input device into which a language is input and determining the input language as a second language; a second control unit 210 that determines whether the first language and the second language are the same language; and warning means for generating a warning when the first language and the second language are different from each other.

Also, the inertial sensor 130 may be provided in the wearable device 100 .

Also, the machine learning unit 330 may perform machine learning based on at least one of an amplitude, a period, and a repeated partial waveform of an operation related to a language input, and may learn using a supervised learning technique.

In addition, the inertial sensor 130 is mounted on the wearable device 100, the wearable device 100 further includes a first communication unit 120 for communication, and the inference unit 230 and the second control unit 210 are It is mounted on the input device, and the input device may further include a second communication unit 220 for communication with the first communication unit 120 .

In addition, the machine learning unit 330 is mounted on the server 300 , and the server 300 may further include a third communication unit 320 for communication with the second communication unit 220 .

In addition, the warning means is at least one of the first warning unit 130 mounted on the wearable device 100 and the second warning unit 240 mounted on the input device.

In addition, the input device is one of the computer 10, the smart phone 20, and the tablet PC.

As another object of the present invention as described above, the machine learning unit 330 machine-learning an operation that occurs when the user 5 inputs a language (S200); and the third communication unit 320 transmitting the machine learning model to the inference unit 210 of the input device (S220); performed in advance or regularly performed, the inertia attached to the body part of the user 5 detecting, by the sensor 130, an operation signal from an operation occurring when the user 5 inputs a language (S100, S110); The first communication unit 120 transmitting the operation signal to the input device (S120); Inferring, by the reasoning unit 230, the input language as the first language based on the operation signal (S130); determining, by the second control unit 210 of the input device, whether the input second language and the inferred first language are the same language (S140); If the first language and the second language are different, the warning means generates a warning ( S150 ).

In addition, the warning step ( S150 ) may further include the step of the second controller 210 converting the second language into the first language.

Also, the first and second languages may be languages selected differently from a group including Korean, English, Japanese, Chinese, Spanish, German, and French.

According to an embodiment of the present invention, the wearable device attached to the user may recognize an input operation, and compare and determine whether the first language inferred according to the input operation is the same as the input second language. Accordingly, the user may be notified of input of an incorrect language only by wearing the wearable device, and it is possible to prevent typos in Korean and English. Even if the user 5 is visually impaired, it is possible to conveniently distinguish between Korean and English on the keyboard.

In addition, it is possible to accurately infer the first language input as the input motion by machine learning the characteristics of the input motion for each language. Since this is based on the structure of the language and the location of the keyboard, the first language can be accurately inferred regardless of the user's gender, age, input speed, etc.

In addition, it is possible to warn the user by determining whether the inferred first language and the input second language are the same. Accordingly, the user can immediately recognize the input of the wrong language, and can correct or change the keyboard.

However, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be able

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical spirit of the present invention together with the detailed description of the present invention to be described later. It should not be construed as being limited only to
1 is a state diagram in which an apparatus for recognizing an input language through machine learning is used in a computer according to an embodiment of the present invention;
2 is a state diagram of using a device for recognizing an input language through machine learning in a smart phone according to an embodiment of the present invention;
3 is a schematic block diagram of an apparatus for recognizing an input language through machine learning according to an embodiment of the present invention;
4A is a graph showing a profile detected by the inertial sensor 130 of the wearable device 100 according to the present invention when Korean is input;
4b is a graph showing a profile detected by the inertial sensor 130 of the wearable device 100 according to the present invention when English is input;
5 is a detailed internal block diagram of the machine learning unit 330, which is a component of the present invention;
6 is a flowchart illustrating a method for recognizing an input language through machine learning according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, since the description of the present invention is merely an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiment described in the text. That is, since the embodiment is capable of various changes and may have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, it should not be understood that the scope of the present invention is limited thereby.

The meaning of the terms described in the present invention should be understood as follows.

Terms such as “first” and “second” are for distinguishing one component from another, and the scope of rights should not be limited by these terms. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component. When a component is referred to as being “connected to” another component, it may be directly connected to the other component, but it should be understood that other components may exist in between. On the other hand, when it is mentioned that a certain element is "directly connected" to another element, it should be understood that the other element does not exist in the middle. On the other hand, other expressions describing the relationship between elements, that is, "between" and "between" or "neighboring to" and "directly adjacent to", etc., should be interpreted similarly.

The singular expression is to be understood as including the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" or "have" refer to the specified feature, number, step, action, component, part or these It is intended to indicate that a combination exists, and it should be understood that it does not preclude the possibility of the existence or addition of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. Terms defined in a commonly used dictionary should be interpreted as having the meaning consistent with the context of the related art, and cannot be interpreted as having an ideal or excessively formal meaning unless explicitly defined in the present invention.

embodiment composition

Hereinafter, the configuration of the preferred embodiment will be described in detail with reference to the accompanying drawings. 1 is a state diagram of a computer using an apparatus for recognizing an input language through machine learning according to an embodiment of the present invention, and FIG. 2 is an apparatus for recognizing an input language through machine learning according to an embodiment of the present invention. It is a usage state diagram in use in a smartphone. 1 and 2 , a user wears a wearable device 100 and inputs a language.

In the present invention, the wearable device 100 includes electronic devices provided in clothes, hats, shoes, handbags, bags, etc. as well as watches, rings, glasses, necklaces, and headphones. The wearable device 100 includes an independent power source (eg, a secondary battery), a display (eg, a touch screen or LED), and an input device (button, touch, etc.). In the present invention, a smart watch is shown and described as an embodiment of a wearable device. The wearable device 100 is worn on the wrist of the user 5 .

The input device may be the keyboard 14 of the computer 10 or the keyboard application 22 of the smartphone 20 .

3 is a schematic block diagram of an apparatus for recognizing an input language through machine learning according to an embodiment of the present invention. As shown in FIG. 3 , the wearable device 100 includes a first control unit 110 , a first communication unit 120 , and an inertial sensor 130 .

The first control unit 110 may be a microcomputer, a CPU, or an AP (application processor).

The first communication unit 120 may perform two-way data communication with the input device, and may be wired or wireless. The first communication unit 120 may be a wireless module such as Wi-Fi, NFC, 4G communication network, wireless-LAN, or Bluetooth, and may be a wired communication module such as a USB cable. The first communication unit 120 may transmit an operation signal of the inertial sensor 130 and receive a “warning”.

The inertial sensor 130 may be a gyro sensor, an acceleration sensor, or the like, detects a three-dimensional vibration generated during the user's 5 language input, and outputs it as a time-series motion signal.

The first warning unit 140 is a component for sending a warning to the user when another language is input. The first warning unit 140 may be an LED, an LCD, a speaker, a buzzer, or a vibration generating device of the wearable device 100 .

The input device may be a computer 10 or a smart phone 20 or a tablet PC or a notebook computer. The second control unit 210 may be a microcomputer, a CPU, or an AP (application processor).

The second communication unit 220 may perform two-way data communication with the wearable device 100 , and may perform data communication with the server 300 , and may be wired or wireless. The second communication unit 220 may be a wireless module such as Wi-Fi, NFC, 4G communication network, wireless-LAN, or Bluetooth, and may be a wired communication module such as a USB cable, a LAN line, an optical communication line, or a dedicated line. The second communication unit 220 may receive the operation signal from the first communication unit 120 and transmit a “warning”. The second communication unit 220 receives the machine learning model from the server 300 .

The inference unit 230 is loaded with the model learned by the machine learning unit 330, and when the received motion signal is input, infers what language it is and outputs it.

The second warning unit 140 is a component for sending a warning to the user when another language is input. The second warning unit 240 may be an LED of an input device, an LCD, a speaker, a buzzer, a vibration generating device, and the like.

The server 300 includes a third communication unit 320 , a third control unit 310 , and a machine learning unit 330 . The third communication unit 320 may perform two-way data communication with the input device or the wearable device 100, and may be wired or wireless. The third communication unit 320 may be a wireless module such as Wi-Fi, NFC, 4G communication network, wireless-LAN, or Bluetooth, and may be a wired communication module such as a USB cable, a LAN line, an optical communication line, or a dedicated line. The third communication unit 320 may receive an operation signal from the first communication unit 120 or the second communication unit 220 . The third communication unit 320 transmits the machine learning model to the wearable device 100 or the input device.

The machine learning unit 330 receives a plurality of data sets made of language and motion signals as training data and generates a model by machine learning (deep learning). A supervised learning technique is used as the learning form of model generation. The supervised learning technique is a technique for learning by inputting the motion signal 340 and the language together in the learning training data. As a specific method of the supervised learning technique, methods such as SVM (Support Vector Machine), Naive bayes classifiers, Decision tree, Random forest, etc. can be applied, A feed-forward neural network, a convolutional neural network (CNN), and a recurrent neural network may also be utilized. That is, the machine learning unit 330 extracts the amplitude, period, and repeated partial waveforms and feature points of the operation signal 340 and uses them for deep learning.

The machine learning unit 330 is provided in the server 300 to learn a large amount of data, but may be provided in the computer 10, the smart phone 20, or the wearable device 100 according to selection.

5 is a detailed internal block diagram of the machine learning unit 330, which is a component of the present invention. Although the feedforward neural network is illustrated and described as a model in FIG. 5 , the above-described various techniques may be applied unless the machine learning unit 330 is necessarily limited thereto. As an input of the machine learning unit 330, an operation signal 340 and a language (eg, Korean) are input, and the inferred output is a probability for each type of each language 350 (eg, Korean 95.1%, Mongolian 2.1%, English). 0.1%, etc.). More specifically, '1' is Korean, '2' is English, '3' is Japanese, '4' is Chinese, '5' is Spanish, '6' is German, and '7' is French. Enter The inferred first language 350 becomes the most probable language.

The activation function of the machine learning unit 330 converts the sum of input signals entering individual neurons of the neural network into an output signal, and a rectified linear unit (ReLU) is used as the activation function.

In the affine layer (or affine transform), the dot product operation of a matrix is performed. The model of the machine learning unit 330 may be designed to arrange a plurality of layers of the ReLU layer and the Affine layer as needed.

The Soft Max function infers one language. The softmax function interprets the output of the function as a real number between 0 and 1.0 as a probability of classification.

embodiment movement

Hereinafter, the operation of the preferred embodiment will be described in detail with reference to the accompanying drawings.

4A is a graph showing a time series profile detected by the inertial sensor 130 of the wearable device 100 according to the present invention when Korean is input, and FIG. 4B is a wearable device 100 according to the present invention when English is input. It is a graph showing the time series profile detected by the inertial sensor 130 of the. As shown in FIGS. 4A and 4B , when Korean is input, a unique operation signal unique to Korean is detected according to the structure of a word (eg, Ganadara) and the layout of the keyboard. That is, it can be seen that the Korean input has a larger amplitude than the English input, and this can be separated through machine learning. In addition, it is not necessary to separately input these features from the motion signal, and there is a convenience in using the entire detected signal as learning and training data.

6 is a flowchart illustrating a method for recognizing an input language through machine learning according to an embodiment of the present invention. As shown in FIG. 6 , first, the machine learning unit 330 as shown in FIG. 5 machine learns an operation that occurs when the user 5 inputs a language ( S200 ).

The learned model is transmitted to the inference unit 210 of the input device through the third communication unit 320 (S220). The inference unit 210 stores the model and enters a state in which it can be inferred using the model. This process is performed in advance, and it is repeated regularly or irregularly to keep the model up to date reflecting the user's habits and usage patterns.

Next, the user 5 wears the wearable device 100 on the wrist and inputs a first language (eg, Korean) on the keyboard 140 of the computer 10 ( S100 ). Then, the inertial sensor 130 detects a motion signal from a wrist motion, a finger motion, an impact at the time of typing, etc. generated when the user 5 types ( S110 ).

Then, the first communication unit 120 transmits the operation signal to the input device (S120).

The transmitted motion signal is sent to the reasoning unit 230, and the reasoning unit 230 infers the input language as the first language (eg, Korean) based on the motion signal (S130).

Next, the second control unit 210 of the computer 10 determines whether the actually typed and input second language and the inferred first language are the same language ( S140 ). If the inferred first language is Korean and the input second language is also Korean, it is determined that it is the same language and the word is continuously input without any particular warning.

However, if the first language (eg, Korean) and the second language (eg, English) are different, the warning means generates a warning (S150). This occurs due to an unintentional event such as the user 5 accidentally pressing the language change key, the word processor automatically switching the language, or changing the language to the default language when the program is first started.

In the warning step (S150), the wearable device 100 notifies the user 5 that it is an erroneous input through a warning sound, vibration, flickering of the LCD screen, and the like. The input device notifies the user 5 that it is an incorrect input through a warning sound, a warning text, a screen flickering, and the like.

And, optionally, the second control unit 210 may automatically convert the second language (English) into the first language (Korean) to increase the convenience of the user 5 .

The detailed description of the preferred embodiments of the present invention disclosed as described above is provided to enable any person skilled in the art to make and practice the present invention. Although the above has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art that various modifications and changes can be made to the present invention without departing from the scope of the present invention. For example, those skilled in the art can use each configuration described in the above-described embodiments in a way in combination with each other. Accordingly, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The present invention may be embodied in other specific forms without departing from the spirit and essential characteristics of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention. The present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In addition, claims that are not explicitly cited in the claims may be combined to form an embodiment or may be included as a new claim by amendment after filing.

5: user;
10: input device (computer),
12: monitor,
14: keyboard,
20: input device (smartphone),
22: keyboard application,
100: wearable device,
110: a first control unit;
120: a first communication unit;
130: inertial sensor,
140: first warning unit;
210: a second control unit;
220: a second communication unit;
230: reasoning unit,
240: a second warning unit;
300 : server,
310: a third control unit;
320: a third communication unit;
330: machine learning unit,
340: operation signal,
350: first language.

Claims (11)

an inertial sensor 130 attached to a body part of the user 5 to detect a motion occurring when the user 5 inputs a language;
a machine learning unit 330 for machine learning an operation related to the language input;
an inference unit 230 for inferring the language as a first language based on a learning model from the machine learning unit 330 and an operation signal of the inertial sensor 130;
a second control unit 210 provided in an input device into which the language is input, and configured to determine the input language as a second language;
the second control unit 210 for determining whether the first language and the second language are the same language; and
and warning means for generating a warning when the first language and the second language are different.
The warning means is a first warning unit 130 mounted on the wearable device 100 and a second warning unit 240 mounted on the input device,
The inertial sensor 130 is mounted on the wearable device 100,
The wearable device 100 further includes a first communication unit 120 for communication,
The reasoning unit 230 and the second control unit 210 are mounted on the input device,
The input device further includes a second communication unit 220 for communication with the first communication unit 120,
The machine learning unit 330 performs machine learning by a supervised learning technique based on the amplitude, period, and repeated partial waveforms of the operation related to the language input,
The input device is a computer 10, a smart phone 20, a device for recognizing an input language through machine learning, characterized in that one of a tablet PC. delete delete delete delete The method of claim 1,
The machine learning unit 330 is mounted on the server 300,
The server 300 is an input language recognition apparatus through machine learning, characterized in that it further comprises a third communication unit (320) for communication with the second communication unit (220). delete delete As a recognition method using the input language recognition device according to claim 1 or 6,
The machine learning unit 330 machine learning the amplitude, period, and repeated partial waveforms of the motion generated when the user 5 inputs a language (S200) using a supervised learning technique; and
Step (S220) of the third communication unit 320 transmitting the machine learning model to the inference unit 210 of the input device; is performed in advance or is performed regularly,
detecting, by the inertial sensor 130 mounted on the wearable device 100 attached to the body part of the user 5, an operation signal from the operation generated when the user 5 inputs a language (S100, S110);
The first communication unit 120 transmitting the operation signal to the input device (S120);
Inferring, by the reasoning unit 230, the input language as the first language based on the operation signal (S130);
determining, by the second control unit 210 of the input device, whether the input second language and the inferred first language are the same language (S140);
If the first language and the second language are different, the first warning unit 130 mounted on the wearable device 100 and the second warning unit 240 mounted on the input device generate a warning (S150); including;
The warning step (S150) further includes the step of the second control unit 210 converting the second language into the first language,
The input device is one of a computer 10, a smartphone 20, and a tablet PC,
The first and second languages are languages selected differently from a group including Korean, English, Japanese, Chinese, Spanish, German, and French. delete delete

Images