KR101793607B1 - System, method and program for educating sign language - Google Patents

Abstract

The present invention relates to a sign language education system, a method and a program.
The sign language training method according to an embodiment of the present invention includes: extracting and providing specific item data of a terminal (S100); Extracting reference sign language data corresponding to the item data (S200); Receiving sensing data from at least one body motion measurement device (S300); Combining at least one of the sensing data to generate input sign data (S400); Comparing the input sign language data with the reference sign language data to calculate a comparison result (S500); And performing an evaluation on the item data based on the comparison result (S600).
According to the present invention, the user can receive an evaluation as to whether or not the hydration operation actually performed is accurate, thereby helping improve the hydration skill.

Description

METHOD AND PROGRAM FOR EDUCATING SIGN LANGUAGE BACKGROUND OF THE INVENTION [0001]

The present invention relates to a sign language education system, a method and a program, and more particularly, to a system, a method, and a program for evaluating and training sign language of a user performed according to an item using a body motion measurement device.

Visually impaired people can receive information only through hearing. People with visual impairment can obtain information only through the metabolism or sound effect of a person in a visual medium such as TV. This is because information such as motion and behavior is unknown. To solve this problem, a broadcasting station can provide services such as screen commentary broadcasting.

In addition, hearing-impaired people communicate through texts on computers or on texts using visual sign language. However, there are many cases in which communication through texts can not bring meaning sufficiently due to differences in grammar and expression systems. This is because communication through text causes problems such as information distortion and loss.

Communication between the speaker and the non - speaker is dominated by the way through the sign language interpreter and the way through the text. However, the method through the interpreter is realistically limited in cost. Therefore, when communicating with the hearing impaired, simple forms of text are used as the main medium. Because of differences in sign language and Korean grammar and expressive system, signifiers often fail to convey meaning when conveying meaning to text.

People who need to communicate with people who need to use sign language (for example, hearing impaired people) try to learn sign language. However, sign language has many differences (eg, differences in word order) from the grammatical system of the general language (ie, the language of conversation, not sign language). Therefore, people have difficulty learning.

In addition, the sign language may not communicate correctly if the operation is not accurate, and communication may not be achieved if the operation is correct but does not match the sign language order.

Therefore, a sign language education system and method for acquiring sign language data performed by a user for a specific question presented and judging whether or not they match the standard sign language data according to the presented question so that the user can learn an accurate sign language word and word order And programs.

The sign language training method according to an embodiment of the present invention includes: extracting and providing specific item data of a terminal; Extracting reference sign language data corresponding to the item data; Receiving sensing data from one or more body motion measurement devices; Combining at least one of the sensing data to generate input signatures; Comparing the input sign language data with reference sign language data and calculating a comparison result; And performing an evaluation of the item data based on the comparison result.

The reference saliency data extracting step extracts reference saliency data stored in the terminal database in a manner matching with the question data.

If the item data corresponds to a sentence, the step of extracting the reference sign language data includes: changing a word order of the item data into a sign language word; And searching for and matching matched data corresponding to each word of the question data.

In addition, when the body motion measuring device includes the glove type measuring device and the attachable type measuring sensor device, the sensing data receiving step may include receiving finger and wrist sensing data from the glove type measuring device; And receiving sensing data from a measurement sensor device attached to each body unit.

The sensing data receiving step may include: requesting a user to perform a specific reference posture or a reference motion; And determining an initial position of each of the body motion measuring apparatuses according to the reference posture or the reference movement.

The input sign language data generation step may include tracking the movement of the body unit based on the initial position, and calculating a positional relationship between the left hand and the right hand.

Receiving the image obtained by the vision sensor device when the body motion measuring device includes a vision sensor device, the image including both hands of a user; And recognizing a positional relationship between the left and right hands in the image.

The step of performing the evaluation may include calculating a matching rate between the reference sign language data and the input sign language data of a word corresponding to the item data when the question data is a word.

The evaluating step may include the step of comparing the at least one reference word data of the reference sign language data and the at least one input word data of the input sign language data when the item data is a sentence, ; Matching the input word data having the highest match rate with the reference word data; Calculating a word match result based on a match rate for each of the input word data; And the distance difference is calculated by accumulating the difference in distance between the input word data and the reference word data to calculate a word order match result, wherein the distance difference is shifted so that the specified input word data is arranged in the same sentence position as the reference word data having the highest match rate The number of words; And calculating an evaluation score by reflecting the word match result and the word match result.

The method may further include generating feedback data for the user based on the word matching result and the word matching result.

The method may further include determining difficulty level of the next item data by reflecting the evaluation score.

The sign language education program according to another embodiment of the present invention is combined with a hardware terminal and executes the sign language training method and is stored in a medium.

According to the present invention as described above, the following various effects are obtained.

First, the user can be evaluated whether the actual hydration operation is correct, which can help improve the hydration ability.

Second, through the present invention, the user can evaluate not only the accuracy of the operation on the word but also the accuracy of the sign language of the specific sentence. Therefore, the user can accurately learn the sign language order different from the Korean word order.

Thirdly, according to the present invention, if only a database of sign language different for each language is added, the user can learn the sign language of various countries through one terminal.

Fourth, since the positional relationship between the left hand and the right hand of the user can be obtained by using the vision sensor or tracking the movement of the body motion measuring device from the initial position, It is possible to evaluate whether it is correct or not.

Fifth, it is possible to construct a variety of item data, so that it is possible to perform hydration item learning necessary for the user's situation. In addition, various contents can be combined and provided, so that the user can enjoy learning the sign language happily. In addition, through the evaluation of sign language, questions appropriate to the level of the user's learning can be presented, and the level-specific learning can be provided to the user.

1 is a block diagram of a sign language training system according to an embodiment of the present invention.
2 is a flowchart of a sign language training method according to an embodiment of the present invention.
3 is a flowchart illustrating a process of evaluating sentence item data according to an embodiment of the present invention.
4 is an exemplary diagram for calculating a word order match result based on a match rate of a word according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms " comprises "and / or" comprising "used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated element.

In this specification, the body motion measuring apparatus corresponds to an apparatus for sensing movement of the user's body and transmitting motion data to the terminal through wire / wireless communication. The body motion measurement device may be in the form of a wearable device, or it may be a sensor device that can be attached to various positions of the body. In addition, the body motion measuring device may be various types of devices capable of measuring the user's body motion. For example, when the body motion to be measured is the movement of the hand, the body motion measuring device can be implemented as a hand wearing type to be worn on the hand. Further, for example, when the body movement to be measured is movement of both arms, the body motion measuring apparatus may be a sensor patch or the like that can be attached to each body region. In addition, when collectively measuring movement of various body parts, the body motion measuring device may include various measuring devices such as a hand-held (e.g., glove-type) measuring device, a body-mounted measuring sensor device, and the like.

Hereinafter, a sign language education system, a method, and a program according to embodiments of the present invention will be described with reference to the drawings.

1 is a block diagram of a sign language training system according to an embodiment of the present invention.

The sign language education system according to an embodiment of the present invention is implemented by the internal configuration of the terminal 100. [ The terminal 100 can be divided into a mobile terminal 100 and a fixed terminal 100 according to whether the mobile terminal 100 can be moved or not. For example, the terminal 100 may include a cellular phone, a PCS phone, a synchronous / asynchronous IMT-2000 (International Mobile Telecommunication-2000) service corresponding to the mobile terminal 100, A PDA (Personal Digital Assistant), a smart phone, a WAP phone (Wireless Application Protocol), a mobile game machine, a tablet PC , A netbook, a notebook (Note Book), and the like, and may correspond to a desktop PC, a television, and the like corresponding to the fixed terminal 100.

The terminal 100 includes a control unit 110; A communication unit 130; And an output unit 130, all or part of which are shown. The terminal 100 is not limited to the above-described components, and may further include additional components.

The controller 110 typically controls the overall operation of the terminal 100. For example, data communication, image processing for playback on the display, and body motion estimation (e.g., input mood data evaluation based on user's body movements), and the like. Various functions performed by the controller 110 will be described later.

The communication unit 130 receives sensing data from the body motion measurement device 200. In addition, the communication unit 130 transmits the received sensing data to the control unit 110. Also, the communication unit 130 may transmit the output based on the evaluation result calculated on the basis of the sensing data to the body motion measurement apparatus 200.

The communication unit 130 includes a wireless communication unit for receiving the motion data from the body motion measurement apparatus 200 through a wired communication unit connected to the body motion measurement apparatus 200 and receiving data, can do. The wireless communication unit may include a wireless Internet module; Or a short range communication module.

The wireless Internet module refers to a module for wireless Internet access, and may be built in or externally attached to the terminal 100. Wireless Internet technologies include WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), LTE (Long Term Evolution-Advanced) or the like can be used.

The short-range communication module is a module for short-range communication. (Bluetooth), BLE (Bluetooth Low Energy), Beacon, RFID (Radio Frequency Identification), NFC (Near Field Communication), Infrared Data Association (IrDA) UWB (Ultra Wideband), ZigBee, NRF and the like can be used.

The output unit 130 outputs information to be provided to the user. The output unit 130 may include a display unit, an audio output unit, and the like.

The display unit performs a function of displaying (outputting) information processed in the terminal 100. Also, the display unit may be implemented with a touch screen by being combined with the touch sensor. The display unit can receive an input operation from a user through a touch operation. For example, the terminal 100 can select a question to be evaluated by receiving the touch operation of the user at a point corresponding to specific item data in the item list displayed on the screen.

The various embodiments described herein may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

In addition, the terminal 100 may further include a memory. The memory may store a program for the operation of the control unit 110 and may store data to be input / output or data (e.g., training data for receiving and storing motion data, etc.) It is possible. The memory may be included in the controller 110.

The memory may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), a RAM access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM) And an optical disc. The mobile terminal 100 may operate in association with a web storage that performs the storage function of the memory on the Internet.

According to a hardware implementation, the embodiments described herein may be implemented as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays , Microprocessors, microprocessors, microprocessors, and other electronic units for carrying out other functions. In some cases, the embodiments described herein may be implemented by the controller 110 itself.

According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein.

The software code may be implemented in a software application written in a suitable programming language. The software code may be stored in a memory and executed by the control unit 110. [

2 is a flowchart of a sign language training method according to an embodiment of the present invention.

Referring to FIG. 2, a sign language training method according to an embodiment of the present invention includes: extracting and providing specific item data of a terminal 100 (S100); Extracting reference sign language data corresponding to the item data (S200); Receiving sensing data from at least one body motion measurement device (200) (S300); Combining at least one of the sensing data to generate input sign data (S400); Comparing the input sign language data with the reference sign language data to calculate a comparison result (S500); And performing an evaluation on the item data based on the comparison result (S600). A sign language training method according to an embodiment of the present invention will be described in order.

The terminal 100 extracts and provides specific item data (S100). That is, the terminal 100 extracts specific item data according to the item selection request received from the user. For example, when the terminal 100 receives a specific operation from the user's body motion measurement apparatus 200 (for example, when the body motion measurement apparatus 200 is a hand-held type apparatus) The operation of collecting hands after a hand operation indicating the operation of the user) can be performed to select the item data. In addition, when the display unit is a touch screen, the terminal 100 can display various item data lists on the screen, and can receive and select a touch operation for specific item data from the user. Then, the terminal 100 may provide the selected item data to the user through the display unit or the sound output unit.

The terminal 100 extracts reference sign language data corresponding to the item data (S200). The reference sign language data corresponds to a sign language expression in which the meaning is matched with the specific question data, which is data to be compared with the user sign language operation (that is, input sign language data) obtained by the body motion measurement apparatus 200. The terminal 100 may extract reference sign language data corresponding to the item data in various ways. However, the reference sign language data extraction method is not limited to the method described below, and various methods can be applied.

In one embodiment of the standard hydration data extraction method, the terminal 100 can extract reference hydration data stored in an internal database and matched with specific item data. That is, the terminal 100 may store hydration operation data (i.e., reference hydration data) corresponding to specific item data in the memory. The terminal 100 may store a sign language operation expression corresponding to the item data that is a specific word or a simple expression (e.g., a personnel expression), and may store a sign language operation expression corresponding to the item data that is a specific sentence. If the item data is a sentence, the terminal 100 may store the reference sign language data generated in advance in the database in accordance with the order of sign language.

Also, since the sign language expressions for item data having the same meaning may be different for each country, the terminal 100 may match and store country-specific sign language expressions for the same item data in the database. The terminal 100 can extract and present the sign language expression of a specific country set by the user as the standard sign language data.

In another embodiment of the reference saliency data extraction method, when the item data corresponds to a sentence, the terminal 100 can directly generate the reference saliency data corresponding to the selected or extracted item data. That is, the step of extracting the reference hydration data (S200) comprises the steps of: changing the order of the item data into a sign language order; And searching for and matching matched data corresponding to each word of the question data. The terminal 100 includes data related to the numerical ordering rule and can convert the item data (i.e., textual sentences) extracted on the basis of the ordering rule data into a numerical order. The terminal 100 may first rearrange the sentence in units of phrases in order to rearrange the phrases constituting the sentence in the sign language order. Segmentation in units of words can be performed based on the spacing of sentences. After that, the terminal 100 can search each database in the database to load corresponding sign language expressions, and connect one or more sign language expressions to generate reference sign language data.

The terminal 100 receives the sensing data from at least one body motion measuring device 200 (S300). The at least one body motion measuring apparatus 200 may include a glove measuring apparatus, a body attaching measuring sensor apparatus, a vision sensor apparatus, and the like. The glove-type measuring device can be worn on both hands of the user. In addition, the attachment type measurement sensor device may be attached to both the lower arm (i.e., from the elbow to the wrist) or the upper arm (i.e., the portion from the shoulder to the elbow), which is the user's body part used for hydration. For example, the body attachment type sensor device can be attached to the upper arm and the lower arm, respectively, so that the bending state, the motion state, and the positional relationship of the arm can be measured.

When the body motion measuring apparatus 200 includes a hand-held measuring apparatus and an attaching-type measuring sensor apparatus, the sensing data receiving step S300 is a step of receiving finger movement and wrist movement from the hand- Receiving sensing data; And receiving sensing data from the attached type measurement sensor device attached to each of the body units. The hand-held measuring device can be worn on both hands of a user and can measure the bending state or motion of each finger, the bending or movement of the wrist (e.g., the direction the palm is heading with the wrist rotation, etc.) . The hand-held measuring device can measure the state or movement of the finger and wrist, including a banding sensor or an IMU sensor.

In addition, the attachment type measurement sensor device can include an inertial sensor, so that it can be attached to each of the respective body units to calculate motion. That is, the attached type measurement sensor device attached to the upper arm and the lower arm acquires sensing data through the inertial sensor and transmits the sensed data to the terminal 100. The terminal 100 senses the positional relationship between the upper arm and the lower arm Can be calculated. Also,

In addition, when the body motion measuring apparatus 200 includes the vision sensor apparatus, the vision sensor apparatus can acquire image data to be measured by the terminal 100 as movement data of the hands and arms, as sensing data. In addition, a hand-held measurement device or an attachment-type measurement sensor device can include an identification mark that can be recognized by the vision sensor device, and the vision sensor device can measure the movement of one or more identification markers and transmit them to the terminal 100 . To this end, an embodiment of the present invention is a method comprising: receiving an image obtained by the vision sensor device, the image including both hands of a user; And recognizing a positional relationship between the left and right hands in the image. The sensing data obtained by the vision sensor device can be used for calculation of the positional relationship between both hands of the user and both hands by the terminal 100 as described later.

The vision sensor device may be formed of a necklace type, a deck type disposed on the front side of the user, an inner mounting type of the terminal 100, or the like. When the vision sensor device is of the necklace type, the vision sensor device can acquire images of the movements of both hands and both arms in front of the user's body. In addition, when the vision sensor is deck type or terminal 100 attached or included, it is disposed in front of the user at the input of the sign language representation (i.e., at the input sign language data input) for the item data to include both hands and arms To obtain a frontal image.

The terminal 100 combines one or more of the sensing data to generate input sign data (S400). That is, the terminal 100 may combine the sensing data obtained by the at least one body motion measuring apparatus 200 to generate the motion of the whole body part attached with the body motion measuring apparatus 200 as input signatures. For example, the terminal 100 can detect the shape of an arm at each time point (for example, a positional relation between the upper arm and the lower arm corresponding to the bending state of the arm) through the sensing data measured from the attached type measurement sensor device attached to both arms The arrangement state of each arm, and the movement form of each arm) can be calculated. In addition, the terminal 100 can acquire the shape of the user's hand at each time point through sensing data received from the hand-held type measurement sensor device (for example, each finger bending state, wrist bending state, wrist rotation state, Can be calculated. Then, the terminal 100 may combine the arrangement state of each arm with the shape of each hand to generate a full hydration expression (i.e., input hydration data) performed by the user.

When the terminal 100 receives the sensing data (i.e., image data) obtained by the vision sensor device, the terminal 100 analyzes the sensing image obtained through the vision sensor and determines the positional relationship between the two hands or both arms Can be calculated. The terminal 100 may generate a full hydration expression (i.e., input hydration data) performed by the user by combining the positional relationship of the two hands or both arms calculated through the image together with the shape of the arm and the shape of the hand.

As described later, when the terminal 100 receives the reference posture or the result of performing the reference motion from the user, the terminal 100 calculates the reference position or the reference motion based on the reference position or the reference position , It is possible to calculate the relative position of each measurement point (for example, a body point on which the body-mounted type measurement sensor device is attached) with respect to the initial position of each viewpoint in performing the user's sign language expression. Accordingly, the terminal 100 can calculate the positional relationship of both hands or both arms based on the relative positions of both hands or both arms with respect to the reference position at the same time point, and can combine with the shape of the arm and the shape of the hand, (I.e., input signatures data) to be generated.

The terminal 100 compares the inputted sign language data with the reference sign language data to calculate a comparison result (S500). That is, the terminal 100 can calculate the result by comparing the reference sign language data (that is, the reference sign language sign) corresponding to the item data and the generated input sign language data (that is, the sign language sign performed by the user) . The reference saliency data may correspond to one or more data that previously measured the reference user's motion for the sign language operation corresponding to the word, and may include an axial acceleration sensor value, a gyro sensor value, a geomagnetism sensor value, (E.g., data values of the position, inclination, direction, etc. of a specific body part in the image obtained through the vision sensor device), and the like. Also, the terminal accumulates the hydration operations received from the user as input hydration data rather than a specific one of the sensed data, and performs a machine learning classification technique such as statistical classification, supporting vector machine, decision tree, To generate a specific value or a specific range corresponding to the reference sign language data.

In one embodiment of the method of comparing the reference hydration data and the input hydration data, the terminal may calculate and compare the difference or distance between the two data with respect to changes over time of the input hydration data and the reference hydration data. For example, the terminal can receive the reference hydration data through the operation of the actual person wearing the measurement sensor device, and can store the sensing value of the measurement sensor devices corresponding to each time point. Thereafter, the terminal may calculate the difference by comparing the sensing value of each measurement sensor device according to input sign language data with the sensing value of each measurement sensor device according to the reference sign language data. In addition, the input signatures and the reference signatures can be changed by changing the domain for data analysis (for example, by changing to the frequency domain) to calculate the difference. Also, the terminal can compare the reference sign language data with the input sign language data by applying a DTW (Dynamic Time Wapping) algorithm. However, the method of comparing the reference sign language data with the input sign language data is not limited to this, and various other methods can be applied.

In a case where the item data is a word or a short expression (for example, a tolerable expression such as a greeting), the terminal 100 may calculate the reference value of the word corresponding to the item data The matching rate between the sign language data and the input sign language data can be calculated. That is, since the item data corresponds to one sign language expression and the separate word order expressions do not coincide, the coincidence rate of the operation can be calculated by comparing the reference sign language data with the input sign language data on a one-to-one basis.

If the item data is a sentence, the terminal 100 may compare one or more reference word data of the reference sign language data and one or more input word data of the input sign language data (step S500) Respectively, and calculating a matching rate. The user can input the sign language operation in a different order from the word order corresponding to the item data. In order for the terminal 100 to evaluate the accuracy of the word order, it is necessary to identify which word in the input sign language data corresponds to each word of the reference saliency data. To this end, the terminal 100 divides the reference sign language data and the input sign language data in units of phrases (i.e., generates one or more reference word data and one or more input word data), and transmits all the reference word data and all input word data to each other The matching rate can be calculated. The terminal 100 may perform input / output data matching the specific reference word data in the evaluation step S600, which will be described later, through the calculated matching ratio by performing the mutual comparison between the words.

In an embodiment of a method of dividing input sign language data in units of words, a terminal receives a signal according to an operation on an electrical switch, a mechanical switch, or a pressure sensor provided or attached to a body motion measurement device from a user, Termination can be recognized. For example, the user can input the end of the word to the terminal by operating a switch provided at a specific position of the glove-type measuring device (for example, the detection area of the glove-type measuring device that can be easily manipulated with the thumb).

In another embodiment of the method of dividing the input sign language data in units of words, when the user takes a predetermined gesture defined by the user, the terminal recognizes the gesture and recognizes the end of the word. According to another embodiment of the method for dividing the input sign language data in units of words, if the terminal performs the operation for a specific word and maintains the predetermined time or longer, the terminal can recognize that the specific word is ended. Further, in another embodiment of a method of dividing input sign language data into units of words, when a terminal exceeds a time range generally performed for a specific word (or word), it is recognized that a specific word is terminated and the next word is performed can do.

The terminal 100 performs evaluation of the item data based on the comparison result (S600). When the item data is a word or a short expression (for example, a generic expression such as a greeting), the terminal 100 can calculate a score that matches the comparison result (that is, the matching rate of the reference word data and the input word data) have.

If the item data is a sentence, the terminal 100 evaluates the degree of word order matching based on the result of matching rate by performing a comparison between the reference word data and the input word data, Can be evaluated. That is, one embodiment of the evaluation performing step S600 may include matching (S610) the input word data having the highest matching rate with the reference word data as shown in FIG. 3; And the distance difference is calculated by accumulating the difference in distance between the matched input word data and the reference word data to calculate a word order match result, wherein the distance difference is calculated based on the reference word data having the highest match rate (i.e., matched reference word data) (S620), which is the number of phrases to be moved to be placed in the same sentence position as the sentence; A step (S630) of calculating a word matching result based on a matching rate of each of the input word data; And calculating an evaluation score by reflecting the word match result and the word match result (S640).

First, the terminal 100 may match the input word data having the highest matching rate with the reference word data based on the comparison result between the reference word data and the input word data (S610). For example, as shown in FIG. 4, an input sentence corresponding to the input sign language data is compared with the phrase data of the reference sentence corresponding to the reference sentence data to calculate a match rate, and a criterion having the highest match rate (Ie, the input word 1 is the reference word 3, the input word 2 is the reference word 2, and the input word 3 is the reference word 1). Thereafter, the terminal 100 may calculate the word order match result by accumulating the difference in distance between the matched input word data and the reference word data (S620). The distance difference may be a number of phrases in which a specific input word data is shifted to be placed in the same sentence position as the reference word data having the highest matching rate (i.e., matched reference word data). Since the input word data having the highest concordance rate with the reference word data is likely to be the expression intended by the user as the reference word data, the terminal 100 calculates the distance between the reference word data having the highest concordance rate and the input word data The error rate of the word order or the concordance rate of the word order can be calculated. For example, as shown in FIG. 4, the input word 3 is shifted by two words to be placed at the position of the reference word 1, and the input word 1 is shifted by two words to be placed at the position of the reference word 3, The result of word order matching that is the number of shifts (shifting times) may correspond to 4. Thereafter, the terminal 100 may calculate a word match result based on the match rate for each of the input word data (S630). That is, the terminal 100 may calculate the word matching result by reflecting the matching rate of the input word data and the reference word data that are determined to be mutually coincident. The terminal 100 may include an evaluation criterion according to the matching rate, and thus the word matching result can be calculated. Thereafter, the terminal 100 may calculate the evaluation score by reflecting the word match result and the word match result (S640).

The method may further include generating feedback data for the user based on the word matching result and the word matching result. That is, the terminal 100 may provide the user with feedback (i.e., explanation of the wrong answer) as to what part is wrong in the inputted sign language data inputted by the user. For example, when the input sign language data is compared with the standard sign language data and the word order is incorrect, the terminal 100 may provide a description of the correct word order to the user. If the matching rate between the specific input word data and the matching reference word data is less than a specific value, the terminal 100 can provide the reference word data corresponding to the correct answer on the screen, Motion) can be displayed on the screen.

The method may further include determining difficulty level of the next item data by reflecting the evaluation score. Accordingly, the terminal 100 can sequentially provide the item data corresponding to the level of the user, thereby preventing the user from feeling less interested in sign language learning.

Also, even when the body motion measuring apparatus 200 does not include the vision sensor device, the terminal 100 can calculate the movement of the user's arms or the positional relationship of both hands through the initial position setting method. That is, the initial position of the attachment type measurement sensor device or the hand-held type measurement device is set, and the relative position with respect to the initial position where each device is attached is measured at each time point, have.

In one embodiment, in order to set an initial position, the sensing data reception step (S300) includes: requesting a user to perform a specific reference posture or a reference movement; And determining an initial position of each of the body motion measuring apparatuses 200 according to the reference posture or the reference movement. That is, the terminal 100 may set an initial position (reference position) for receiving a specific reference posture or reference motion from a user and determining a hydration operation thereafter. For example, the terminal 100 may request the user to take a dress posture, and may set the state of the user at the time of performing the dress posture to the initial state.

In an embodiment, in order to calculate a relative position with respect to an initial position of each position, the input sign language data generation step (S400) tracks the movement of the body unit on the basis of the initial position, The positional relationship can be calculated. For example, when the attachable type measurement sensor device includes an inertial sensor, it is possible to accumulate the movement of the user measured by the inertial sensor to grasp the state at a specific point in time. That is, the position at a specific time point can be calculated on the basis of the initial position based on the magnitude and direction of acceleration, the tilt angle, and the like, which are measured and accumulated by the measurement sensor device attached to a specific body part. Specifically, the terminal 100 can calculate the relative position of the first point of time (the point at which the relative position is calculated for the first time after calculating the initial position) on the basis of the initial position on the basis of the sensing data measured at the first point of time, The relative position of the time point (the relative position calculation time point after the first time point) can be calculated based on the sensing data measured at the second time point based on the relative position of the first time point.

The above-described sign language training method according to an embodiment of the present invention may be implemented as a program (or an application) to be executed in combination with the terminal 100 as hardware and stored in a medium.

The above-described program may be stored in a computer-readable medium such as C, C ++, JAVA or the like which can be read by the processor (CPU) of the client through the device interface of the client, in order for the terminal 100 to read the program, , A machine language, and the like. The code may include a functional code related to a function or the like that defines the necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the client to execute the functions in accordance with a predetermined procedure can do. In addition, the code may further include a memory reference related code for further information or media needed to cause the processor of the client to perform the functions to be referenced at any location (address) of the client's internal or external memory have. Further, when the processor of the client needs to communicate with any other computer or server that is remote to execute the functions, the code may be transmitted to any remote computer or server using the client's communication module A communication-related code for determining whether to communicate, what information or media should be transmitted or received during communication, and the like.

The medium to be stored is not a medium for storing data for a short time such as a register, a cache, a memory, etc., but means a medium that semi-permanently stores data and is capable of being read by a device. Specifically, examples of the medium to be stored include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, but are not limited thereto. That is, the program may be stored on various recording media on various servers to which the client can access, or on various recording media on the user's client. In addition, the medium may be distributed to a network-connected computer system so that computer-readable codes may be stored in a distributed manner.

According to the present invention as described above, the following various effects are obtained.

First, the user can be evaluated whether the actual hydration operation is correct, which can help improve the hydration ability.

Second, through the present invention, the user can evaluate not only the accuracy of the operation on the word but also the accuracy of the sign language of the specific sentence. Therefore, the user can accurately learn the sign language order different from the Korean word order.

Thirdly, according to the present invention, if only a database of sign language different for each language is added, the user can learn the sign language of various countries through one terminal.

Fourth, since the positional relationship between the left hand and the right hand of the user can be obtained through the vision sensor or the initial measurement position, the present invention can evaluate whether or not the positional relationship between the two hands of the user is correct.

Fifth, it is possible to construct a variety of item data, so that it is possible to perform hydration item learning necessary for the user's situation. In addition, it can be provided in combination with various contents, so that the user can enjoy learning the sign language happily. In addition, through the evaluation of sign language, questions appropriate to the level of the user's learning can be presented, and the level-specific learning can be provided to the user.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

Claims (12)

Extracting and providing specific item data from the terminal;
Extracting reference sign language data corresponding to the item data;
Receiving sensing data from one or more body motion measurement devices;
Combining at least one of the sensing data to generate input signatures;
Comparing the input sign language data with reference sign language data and calculating a comparison result; And
And performing an evaluation on the item data based on the comparison result,
In the evaluation step,
If the item data is a sentence,
The terminal compares one or more reference word data of the reference sign language data with one or more input word data of the input sign language data to calculate a matching rate;
Matching the input word data having the highest match rate with the reference word data;
Wherein the distance difference is calculated by accumulating distance differences between the input word data and the reference word data to calculate a word order match result, wherein the distance difference is shifted so that the specified input word data is arranged in the same sentence position as the reference word data having the highest match rate The number of words;
Calculating a word matching result based on a matching rate of the input word data matched to each of the reference word data; And
And calculating an evaluation score by reflecting the word match result and the word match result. The method according to claim 1,
Wherein the step of extracting the reference sign language-
And extracting the reference sign language data stored in the database in the terminal so as to match the question data. The method according to claim 1,
If the item data corresponds to a sentence,
Wherein the step of extracting the reference sign language-
Changing a word order of the item data into a word order; And
And searching for and matching matched data corresponding to each word of the question data. The method according to claim 1,
When the body motion measuring device includes a glove measuring device and an attachable measuring sensor device,
The sensing data receiving step includes:
Receiving sensory data of finger movements and wrist movements from the glove-type measurement device; And
And receiving sensing data from a measurement sensor device attached to each body unit. 5. The method of claim 4,
The sensing data receiving step includes:
Requesting a user to perform a specific reference posture or a reference motion; And
Determining an initial position of each body motion measurement device in accordance with the reference posture or the reference motion. 6. The method of claim 5,
Wherein the input sign language data generation step comprises:
Wherein the positional relationship between the left hand and the right hand is calculated by tracking the movement of the body unit based on the initial position. 5. The method of claim 4,
If the body motion measuring device includes a vision sensor device,
Receiving an image obtained by the vision sensor device, the image including both hands of a user; And
And recognizing a positional relationship between a left hand and a right hand in the image. The method according to claim 1,
In the evaluation step,
If the item data is a word,
And calculating a matching rate of the input sign language data with the reference sign language data of the word corresponding to the question data. delete The method according to claim 1,
And generating feedback data for the user based on the word match result and the word match result. The method according to claim 1,
And determining the difficulty level of the next question data by reflecting the evaluation score. 11. A sign language training program, coupled to a terminal that is hardware, stored on a medium for carrying out the method of any one of claims 1-8, 10, 11.

Images