WO2017065324A1

WO2017065324A1 - Sign language education system, method and program

Info

Publication number: WO2017065324A1
Application number: PCT/KR2015/010744
Authority: WO
Inventors: 반호영; 최용근; 이수빈; 양동석
Original assignee: 주식회사 네오펙트
Priority date: 2015-10-13
Filing date: 2015-10-13
Publication date: 2017-04-20
Also published as: KR101793607B1; KR20170054198A

Abstract

The present invention relates to sign language education system, method and program. A sign language education method, according to an embodiment of the present invention, comprises: a step (S100) for a terminal extracting particular question data and providing same; a step (S200) for extracting reference sign language data corresponding to the question data; a step (S300) for receiving sensing data items from one or more body movement measurement devices; a step (S400) for combining the one or more sensing data items and generating input sign language data; a step (S500) for comparing the input sign language data and reference sign language data and calculating a comparison result; and a step (S600) for performing evaluation with respect to the question data on the basis of the comparison result. The present invention enables a user's sign language skill to be enhanced by means of evaluation of the accuracy of actual sign language gestures made by the user.

Description

Sign Language Education System, Method and Program

The present invention relates to a sign language education system, method, and program, and more particularly, to a system, method, and program for evaluating and teaching a sign language of a user performed according to an item using a body movement measuring device.

Visually impaired people can only receive information through hearing. The visually impaired can obtain information only through human speech or sound effects in a video medium such as a TV. This is because information such as movement and behavior is not known. In order to solve this problem, broadcasting stations and the like may provide a service such as screen commentary broadcasting.

Hearing impaired people also communicate through sign language that is visually visible or through text on a computer or text on the page. However, communication through texts often does not contain enough meaning due to differences in different grammar and expression systems. This is because communication through texts causes problems such as information distortion and loss.

The communication between the sign language and the non-sign language is mainly done through a sign language interpreter and a text method. However, the method of interpreters is practically limited in cost. Therefore, when communicating with the hearing impaired, a simple form of text is used as the main medium. Due to the difference in sign language and grammar and expression system of Korean, the sign language is often unable to contain the meaning when conveying meaning in text.

People who need to communicate with people who must use sign language (such as hearing impaired people) attempt to learn sign language. However, sign language has many differences (eg, differences in word order) from the grammar of general languages (ie, non-sign language conversational languages). Therefore, people have difficulty learning.

In addition, sign language may not communicate properly if the operation is not correct, and even if the operation is accurate, if the sign language does not match the order may not be communicated.

Accordingly, a sign language education system and method for acquiring sign language data performed by a user with respect to a specific item presented and determining whether or not it matches the standard sign language data according to the presented item, so that the user can learn an accurate sign language vocabulary and word order. And to provide a program.

Sign language education method according to an embodiment of the present invention, the terminal extracts the specific question data provided; Extracting reference sign data corresponding to the item data; Receiving sensing data from at least one body movement measuring device; Generating input sign data by combining one or more pieces of the sensing data; Calculating a comparison result by comparing the input sign language data with reference sign data; And evaluating the item data based on the comparison result.

The extracting of the reference sign data may include extracting the reference sign data matched with the item data stored in the database in the terminal.

In addition, when the item data corresponds to a sentence, the extracting the reference sign language data may include: changing the word order of the item data to a sign language order; And searching and matching sign language data corresponding to each word of the item data.

In addition, when the body movement measuring device includes a glove-type measuring device and the attached measuring sensor device, the sensing data receiving step, receiving the sensing data of the finger movement and wrist movement from the glove-type measuring device; And receiving sensing data from a measurement sensor device attached to each body unit.

In addition, the sensing data receiving step, the step of requesting the user to perform a specific reference posture or standard movement; And determining an initial position of each body movement measuring device according to the reference posture or reference movement.

In addition, the input sign data generating step may be characterized by calculating the positional relationship between the left hand and the right hand by tracking the movement of the body unit on the basis of the initial position.

In addition, when the body movement measuring device includes a vision sensor device, receiving the image obtained by the vision sensor device, the image includes both hands of the user, image receiving step; And recognizing a positional relationship between the left hand and the right hand in the image.

The evaluating step may include calculating a matching rate between the reference sign language data and the input sign data of a word corresponding to the item data when the item data is a word.

In the evaluating step, when the item data is a sentence, the terminal compares one or more reference word data of the reference sign data with one or more input word data of the input sign data, and calculates a matching rate. ; Matching the input word data with the reference word data having the highest matching rate; Calculating a word matching result based on a matching rate with respect to each of the input word data; The word difference result is calculated by accumulating the distance difference between the input word data and the reference word data, wherein the distance difference is moved so that the specific word input data is placed in the same sentence position as the reference word data having the highest matching rate. A word count; And calculating an evaluation score by reflecting the word matching result and the word matching result.

The method may further include generating feedback data for the user based on the word matching result and the word matching result.

The method may further include determining a difficulty level of the next item data by reflecting the evaluation score.

A sign language education program according to another embodiment of the present invention is coupled to a hardware terminal to execute the sign language education method and is stored in a medium.

According to the present invention as described above, has the following various effects.

First, the user can be assessed whether the actual sign language is performed correctly, which can help improve sign language skills.

Second, according to the present invention, the user can be evaluated not only the accuracy of the operation of the word but also the accuracy of the sign language order of a specific sentence. Therefore, the user can learn the sign language order different from the Korean word order.

Third, through the present invention, if only a database of different sign languages is added for each language, the user can learn the sign language of various countries through one terminal.

Fourth, the positional relationship between the user's left hand and right hand can be obtained by using a vision sensor or tracking the movement of the body movement measuring device from the initial position. You can evaluate whether it is correct.

Fifth, it is possible to build a variety of item data to perform sign language learning necessary for the user's situation. In addition, it is possible to combine and provide a variety of content, so that the user can enjoy learning sign language. In addition, through a sign language evaluation, a question suitable for a user's learning level may be suggested to provide a user with level-specific learning.

1 is a block diagram of a sign language education system according to an embodiment of the present invention.

2 is a flowchart illustrating a sign language education method according to an embodiment of the present invention.

3 is a flowchart illustrating a process of evaluating question data, which is a sentence, according to an embodiment of the present invention.

4 is an exemplary diagram for calculating word matching results based on the coincidence rate of words according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various forms, and only the embodiments are intended to complete the disclosure of the present invention, and the general knowledge in the art to which the present invention pertains. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims. Like reference numerals refer to like elements throughout.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used in a sense that can be commonly understood by those skilled in the art. In addition, the terms defined in the commonly used dictionaries are not ideally or excessively interpreted unless they are specifically defined clearly.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components.

In the present specification, the body movement measuring device corresponds to a device for sensing movement of a user and transmitting movement data to the terminal through wired or wireless communication. The body movement measuring device may be in the form of a wearable device, or may be a sensor device that can be attached to various positions of the body. In addition, the body movement measuring device may be a device of various types that can measure the body movement of the user. For example, when the body movement to be measured is a movement of the hand region, the body movement measuring apparatus may be implemented as a wearable type worn on the hand. Also, for example, when the body movement to be measured is the movement of both arms, the body movement measuring device may be a sensor patch or the like that can be attached to each body region. In addition, in the case of comprehensively measuring the movement of various body parts, the body movement measuring device may include various measuring devices such as a wearable type (for example, a glove type) measuring device and a body type measuring sensor device.

Hereinafter, a sign language education system, method, and program according to embodiments of the present invention will be described with reference to the accompanying drawings.

Sign language education system according to an embodiment of the present invention, is implemented by the internal configuration of the terminal 100. The terminal 100 may be divided into a mobile terminal 100 and a fixed terminal 100 according to whether or not it is movable. The terminal 100 may include all types of terminals 100 including the above configuration. For example, the terminal 100 may include a cellular phone, a PCS phone (Personal Communication Service phone), and a synchronous / asynchronous IMT-2000 (International Mobile Telecommunication-2000) corresponding to the mobile terminal 100. ), Mobile terminal 100, Palm Personal Computer (PDA), Personal Digital Assistant (PDA), Smartphone, Smart Phone, WAP phone, Wireless Application Protocao phone, Mobile Game Machine, Tablet PC , A netbook, a notebook (Note Book), and the like, and a desktop PC, a television, and the like corresponding to the fixed terminal 100 may correspond.

The terminal 100 includes a control unit 110; Communication unit 130; And output unit 130; includes all or part. The terminal 100 is not limited to the components described above, and may further include additional components.

The controller 110 typically controls the overall operation of the terminal 100. For example, it performs related control and processing for data communication, image processing for reproduction on the display unit, body movement evaluation (eg, evaluation of input sign data according to user's body movement), and the like. Various functions performed by the controller 110 will be described later.

The communication unit 130 performs a function of receiving sensing data from the body movement measuring apparatus 200. In addition, the communication unit 130 performs a function of transmitting the received sensing data to the control unit 110. In addition, the communication unit 130 may transmit an output according to the evaluation result calculated based on the sensing data to the body movement measuring apparatus 200.

The communication unit 130 includes a wireless communication unit which is connected to the body movement measuring apparatus 200 by wire and receives the movement data from the body movement measuring apparatus 200 through a wired communication unit or a wireless communication method for receiving data. can do. The wireless communication unit wireless internet module; Or a near field communication module.

The wireless internet module refers to a module for wireless internet access and may be embedded or external to the terminal 100. Wireless Internet technologies include Wireless LAN (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), long term evolution (LTE), LTE-A Long Term Evolution-Advanced and the like can be used.

The short range communication module refers to a module for short range communication. Short range communication technologies include Bluetooth, BLE (Bluetooth Low Energy), Beacon, Radio Frequency Identification (RFID), Near Field Communication (NFC), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, NRF, etc. may be used.

The output unit 130 performs a function of outputting information to be provided to the user. The output unit 130 may include a display unit, a sound output unit, and the like.

The display unit displays (outputs) information processed by the terminal 100. In addition, the display unit may be implemented as a touch screen by being combined with a touch sensor. The display unit may receive an input operation from the user through a touch operation. For example, the terminal 100 may select an item to be evaluated by receiving a user's touch manipulation at a point corresponding to the specific item data from the item list displayed on the screen.

Various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware or a combination thereof.

In addition, the terminal 100 may further include a memory. The memory may store a program for the operation of the controller 110 and store input / output data or data generated during the performance evaluation of the body (for example, learning data for receiving and storing movement data). It may be. The memory may be included in the controller 110.

The memory may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g. SD or XD memory, etc.), RAM access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, It may include at least one type of storage medium of the optical disk. The mobile terminal 100 may operate in connection with a web storage that performs a storage function of the memory on the Internet.

According to a hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays (FPGAs). It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. In some cases, the embodiments described herein may be implemented by the controller 110 itself.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

The software code may be implemented as a software application written in a suitable programming language. The software code may be stored in a memory and executed by the controller 110.

2, the sign language education method according to an embodiment of the present invention, the terminal 100 extracts and provides a specific item data (S100); Extracting reference sign data corresponding to the item data (S200); Receiving the sensing data from the at least one body movement measuring device 200 (S300); Generating input sign data by combining one or more pieces of the sensing data (S400); Comparing the input sign language data with reference sign data and calculating a comparison result (S500); And performing an evaluation on the item data based on the comparison result (S600). Sign language education method according to an embodiment of the present invention will be described in order.

The terminal 100 extracts and provides specific item data (S100). That is, the terminal 100 extracts specific item data according to the item selection request received from the user. For example, the terminal 100 may perform a specific operation (for example, when the body movement measuring apparatus 200 is a hand wearable device) from the body movement measuring apparatus 200 from a user, a specific point at which specific item data is displayed on the screen. After the hand pointing operation, the user can select the item data by performing a manipulation operation of collecting the hand). In addition, when the display unit is a touch screen, the terminal 100 may display various item data lists on the screen, and receive and select a touch operation for a specific item data from a user. Thereafter, the terminal 100 may provide the selected item data to the user through the display unit or the audio output unit.

The terminal 100 extracts reference sign data corresponding to the item data (S200). The reference sign data is data performed by comparison with a sign language operation (ie, input sign data) of a user acquired by the body movement measuring apparatus 200 and corresponds to a sign language expression having a meaning consistent with specific item data. The terminal 100 may extract reference sign language data corresponding to the item data in various ways. However, the standard sign language data extraction method is not limited to the method described below, and various methods may be applied.

In one embodiment of the method of extracting the reference sign data, the terminal 100 may extract the reference sign data stored by matching the item data specific to the internal database. That is, the terminal 100 may store sign language operation data (ie, reference sign data) corresponding to specific item data in the memory. The terminal 100 may store a sign language action expression corresponding to the item data, which is a specific word or a simple expression (eg, a greeting expression, etc.), and may store a sign language action expression corresponding to the item data, which is a specific sentence. When the item data is a sentence, the terminal 100 may store reference sign language data generated in advance in order of sign language order in a database.

In addition, since the sign language expression for the item data having the same meaning may be different for each country, the terminal 100 may match and store the sign language for each country with respect to the same item data in the database. The terminal 100 may extract and present a sign language expression of a specific country set by a user as reference sign data.

In another embodiment of the reference sign data extraction method, when the item data corresponds to a sentence, the terminal 100 may directly generate reference sign data corresponding to the selected or extracted specific item data. That is, the reference sign language data extracting step (S200) may include: changing the word order of the item data to a sign language order; And searching and matching sign language data corresponding to each word of the item data. The terminal 100 includes data about a sign language order rule, and may convert item data (ie, a text sentence) extracted based on the word order rule data into a sign language order. In order to rearrange the words constituting the sentence in sign language order, the terminal 100 may first divide the sentence into word units and then rearrange the sentences. Partitioning by word can be performed based on spacing of sentences. Thereafter, the terminal 100 may search each word in a database to load a corresponding sign language expression, and may generate reference sign data by connecting one or more sign language expressions.

The terminal 100 receives sensing data from at least one body movement measuring apparatus 200 (S300). The at least one body movement measuring device 200 may include a glove-type measuring device, a body-mounted measuring sensor device, a vision sensor device, and the like. The glove-type measuring device can be worn on both hands of the user. In addition, the attached measuring sensor device may be attached to both the lower arm (ie, the elbow to wrist) or the upper arm (that is, the shoulder to elbow) which are the body parts of the user utilized for sign language. For example, the body-mounted measuring sensor device may be attached to the upper and lower arms, respectively, to measure the bending state, the movement state, and the positional relationship of the arm.

In addition, when the body movement measuring device 200 includes a hand wearable measuring device and a wearable measuring sensor device, the sensing data receiving step (S300) may include finger movement and wrist movement from the hand worn measuring device. Receiving sensing data; And receiving sensing data from an attached measuring sensor device attached to each body unit. The hand wearable measuring device may be worn on both hands of the user, and may measure the bending state or movement of each finger, the bending or movement of the wrist (for example, the direction of the palm toward the wrist rotation, etc.). . The hand wearable measuring device may include a banding sensor, an IMU sensor, or the like, and measure the state or movement of the finger and the wrist.

In addition, the attachable measuring sensor device may include an inertial sensor, and may be attached to each body unit to calculate a movement. That is, the attached measuring sensor device attached to the upper arm and the lower arm acquires the sensing data through the inertial sensor and transmits the sensing data to the terminal 100, and the terminal 100 has a positional relationship between the upper arm and the lower arm based on the sensing data. Can be calculated. Also,

In addition, when the body movement measuring device 200 includes a vision sensor device, the vision sensor device may obtain image data for measuring the movement of the hand and arm by the terminal 100 as sensing data. In addition, the hand wearable measuring device or the attached measuring sensor device may include an identification mark that can be recognized by the vision sensor device, the vision sensor device measures the movement of one or more identification marks to transmit to the terminal 100. Can be. To this end, an embodiment of the present invention, the step of receiving an image obtained by the vision sensor device, the image includes both hands of the user, the image receiving step; And recognizing a positional relationship between the left hand and the right hand in the image. The sensing data obtained by the vision sensor device may be used to calculate the positional relationship of both arms and hands of the user by the terminal 100, as described below.

The vision sensor device may be formed of a necklace type, a deck type disposed in front of a user, an internal mounting type of the terminal 100, or the like. When the vision sensor device is a necklace type, the vision sensor device may acquire the movement of both hands and arms in front of the user's body as an image. In addition, when the vision sensor is a deck type or a terminal 100 attached or embedded type, it is disposed in front of the user at the time of inputting a sign language expression of the item data (that is, at the time of input sign language data input) to include movement of both hands and arms of the user. A front image can be obtained.

The terminal 100 combines one or more pieces of the sensing data to generate input sign data (S400). That is, the terminal 100 may combine the sensing data acquired by the one or more body movement measuring apparatus 200 to generate the movement of the entire body part to which the body movement measuring apparatus 200 is attached as input sign data. For example, the terminal 100 is the positional relationship between the upper arm and the lower arm corresponding to the shape of the arm (for example, the bent state of the arm) through the sensing data measured from the attached measuring sensor device attached to both arms. , The arrangement of each arm, and the type of movement of each arm) can be calculated. In addition, the terminal 100 determines the shape of the user's hand at each viewpoint through sensing data (for example, each finger bending state, wrist bending state, wrist rotation state, etc.) received from the hand wearable measuring sensor device. Can be calculated. Thereafter, the terminal 100 may generate a total sign language expression (ie, input sign data) performed by the user by combining the arrangement state of each arm and the shape of each hand.

In addition, when the terminal 100 receives the sensing data (ie, image data) acquired by the vision sensor device, the terminal 100 analyzes the sensing image acquired through the vision sensor to determine the positional relationship of both hands or arms. Can be calculated. The terminal 100 may generate a total sign language expression (ie, input sign data) performed by a user by combining the positional relationship of both hands or arms calculated through an image with an arm shape and a hand shape.

In addition, as will be described later, when the terminal 100 receives a reference posture or a standard movement result from the user, the terminal 100 is based on the initial position (or reference position) identified through the reference posture or standard movement. In addition, when performing a sign language expression of a user, a relative position of each measurement point (for example, a body point to which a body-mounted measuring sensor device is attached) may be calculated with respect to the initial position of each time point. Through this, the terminal 100 may calculate the positional relationship of both hands or both arms based on the relative position of both hands or both arms with respect to the reference position at the same time point, combined with the shape of the arm and the shape of the hand by the user. Can generate the entire sign language (ie, input sign data).

The terminal 100 compares input sign language data with reference sign data to calculate a comparison result (S500). That is, the terminal 100 may calculate a result by comparing the reference sign data (ie, a sign language operation as a reference) corresponding to the item data and the generated input sign data (that is, a sign language operation performed by a user). . The reference sign language data may correspond to one or more pieces of data obtained by measuring a reference user's movement for a sign language operation corresponding to the corresponding word in a dictionary.The acceleration sensor value, the gyro sensor value, the geomagnetic sensor value, and the image information value of each axis according to time (For example, data values such as position, tilt, and direction of a specific body part in the image acquired through the vision sensor device). In addition, the terminal accumulates the sign language operations received from the user as input sign data from the user instead of one specific sensing data, and uses a statistical classification method or a machine learning classification method such as supporting vector machine, decision tree, or deep learning. Learning may be performed to generate a specific value or a specific range corresponding to the reference sign data.

In one embodiment of the method of comparing the reference sign data and the input sign data, the terminal may calculate and compare the difference or distance between the two data with respect to the change of the input sign data and the reference sign data over time. For example, the terminal may receive reference sign data through the operation of a real person wearing a measurement sensor device and store sensing values of the measurement sensor devices corresponding to each time point. Thereafter, the terminal may calculate a difference by comparing the sensing value of each measurement sensor device according to the input sign data with the sensing value of each measurement sensor device according to the reference sign data. In addition, the difference between the input sign data and the reference sign data may be calculated by changing a domain for data analysis (for example, changing the frequency domain). In addition, the terminal may compare the reference sign data with the input sign data by applying a dynamic time wafer (DTW) algorithm. However, the method of comparing the reference sign data and the input sign data is not limited thereto, and various methods such as the present invention may be applied.

In one embodiment of the comparison result calculation step (S500), if the item data is a word or short expression (for example, common expressions such as greetings), the terminal 100 is based on the word corresponding to the item data A coincidence rate between the sign language data and the input sign data can be calculated. That is, since the item data corresponds to one sign language expression and a separate word order expression does not coincide, it is possible to calculate the match ratio of the operation by comparing the reference sign data and the input sign data one to one.

In another embodiment of the comparing result calculating step (S500), when the item data is a sentence, the terminal 100 is one or more reference word data of the reference sign data and one or more input word data of the input sign data Comparing with each other, calculating a matching rate; may include. The user may input a sign language operation in a different order from the word order corresponding to the item data. In order to evaluate the accuracy of word order, the terminal 100 must identify which word in the input sign data corresponds to each word of the reference sign data. To this end, the terminal 100 divides the reference sign data and the input sign data into word units (ie, generates one or more reference word data and one or more input word data), and mutually crosses all the reference word data and all the input word data. By comparison, the coincidence rate can be calculated. The terminal 100 may determine the input word data matching the specific reference word data in the evaluation performing step S600 described later through the matching rate calculated by performing mutual comparison between words.

In an embodiment of dividing the input sign data into word units, the terminal receives a signal according to an operation of an electrical switch, a mechanical switch, or a pressure sensor provided or attached to a body movement measuring device from a user. You can recognize the closure. For example, a user may input the end of a word to a terminal by operating a switch provided at a specific position of the glove-type measuring device (for example, the detection area of the glove-type measuring device which can be easily operated with the thumb).

In addition, as another embodiment of a method of dividing the input sign data into word units, when the user takes a predetermined specific gesture, the terminal may recognize the end of the word by recognizing it. In addition, as another embodiment of a method of dividing the input sign data into word units, the terminal may recognize that a specific word is over when the operation for a specific word is maintained for a predetermined time. In another embodiment of the method of dividing the input sign data into word units, the terminal recognizes that a particular word is terminated and the next word is performed when the time range generally performed for each word (or word) is exceeded. can do.

The terminal 100 evaluates the item data based on the comparison result (S600). When the item data is a word or a short expression (for example, common expressions such as greetings), the terminal 100 may calculate a score corresponding to a comparison result (that is, a match rate between the reference word data and the input word data). have.

In addition, when the item data is a sentence, the terminal 100 evaluates the degree of word order matching based on the result of the matching rate through mutual comparison between the reference word data and the input word data, and evaluates the degree of word match between words. Can be evaluated That is, one embodiment of the evaluation performing step (S600), as shown in Figure 3, the step of matching the input word data and the reference word data with the highest matching rate (S610); Computing a word order result by accumulating the distance difference between the matched input word data and the reference word data, wherein the distance difference is a reference word data (ie, matched reference word data) having the highest matching rate. Step S620, which is the number of words moved to be placed in the same position as the sentence; Calculating a word matching result based on a matching rate with respect to each of the input word data (S630); And calculating an evaluation score based on the word matching result and the word matching result (S640).

First, the terminal 100 may match the input word data and the reference word data having the highest matching rate based on the comparison result between the reference word data and the input word data (S610). For example, as shown in FIG. 4, a match rate is calculated by comparing word phrase data of an input sentence corresponding to the input sign data with a reference sentence corresponding to the reference sign data, and a criterion having the highest match rate for each input phrase. A word (ie, input word 1 may be a reference word 3, input word 2 may be a reference word 2, and input word 3 may be a reference word 1). Thereafter, the terminal 100 may calculate the word matching result by accumulating the distance difference between the matched input word data and the reference word data (S620). The distance difference may be the number of words that are moved so that the specific input word data is placed in the same sentence as the reference word data having the highest matching rate (that is, the matched reference word data). Since the input word data having the highest matching rate with the reference word data is most likely to be an expression intended by the user to input the reference word data, the terminal 100 determines the distance difference between the reference word data with the highest matching rate and the input word data. By calculating the error degree of the word order or the coincidence rate of the word order can be calculated. For example, as shown in FIG. 4, input word 3 is moved by two words to be placed in place of reference word 1, and input word 1 is moved by two words to be placed in place of reference word 3. The word match result, which is the number of shifts (shifting), may correspond to four. Thereafter, the terminal 100 may calculate a word matching result based on a matching rate with respect to each of the input word data (S630). That is, the terminal 100 may calculate a word matching result by reflecting a match rate between the input word data and the reference word data that are determined to match each other. The terminal 100 may include an evaluation criterion according to the coincidence rate, thereby calculating the word matching result. Thereafter, the terminal 100 may calculate an evaluation score by reflecting the word matching result and the word matching result (S640).

The method may further include generating feedback data for the user based on the word matching result and the word matching result. That is, the terminal 100 may provide the user with feedback (that is, an incorrect answer commentary) regarding which part of the input sign data input by the user is wrong. For example, if the input sign data is wrong in comparison with the reference sign data, the terminal 100 may provide a description of the correct answer order to the user. In addition, when the matching rate between the specific input word data and the reference word data matched to it is equal to or less than the specific value, the terminal 100 may provide the reference word data corresponding to the correct answer on the screen, It can be explained by displaying on the screen whether the motion is wrong.

The method may further include determining a difficulty level of the next item data by reflecting the evaluation score. Through this, the terminal 100 may provide the item data corresponding to the user's level in order, thereby preventing the user from becoming less interested in the sign language learning.

In addition, even if the body movement measuring device 200 does not include a vision sensor device, the terminal 100 may calculate the movement of both arms of the user or the positional relationship of both hands through the initial position setting method. That is, the initial position of the attached measuring sensor device or the hand wearable measuring device can be set, and the relative position with respect to the initial position to which the device is attached can be measured at each time point to determine the movement of the user's arms or the positional relationship between the two hands. have.

In one embodiment, to set the initial position, the sensing data receiving step (S300), the step of requesting the user to perform a specific reference posture or reference movement; And determining an initial position of each body movement measuring apparatus 200 according to the reference posture or the standard movement. That is, the terminal 100 may set an initial position (reference position) for determining a sign language operation after receiving a specific reference posture or reference movement from the user. For example, the terminal 100 may request a dressing posture from the user, and thus may set the state at the time of performing the dressing posture of the user to the initial state.

In addition, in one embodiment, to calculate the relative position with respect to the initial position of each position, the input sign data generating step (S400), by tracking the movement of the body unit based on the initial position, the left and right hand The positional relationship can be calculated. For example, when the attached measurement sensor device includes an inertial sensor, the state of a specific point in time may be determined by accumulating the movement of the user measured by the inertial sensor. That is, the position of the specific time point may be calculated based on the initial position based on the magnitude and direction of the acceleration and the inclination angle accumulated and measured by the measurement sensor device attached to the specific body part. Specifically, the terminal 100 may calculate the relative position of the first time point (the time of calculating the relative position for the first time after calculating the initial position) based on the initial position based on the sensing data measured at the first time point, and the second position. The relative position of the viewpoint (relative position calculation point after the first viewpoint) may be calculated based on the sensing data measured at the second viewpoint based on the relative position of the first viewpoint.

Sign language teaching method according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with the terminal 100 which is hardware and stored in a medium.

The above program is C, C ++, JAVA that the client's processor (CPU) can be read through the client's device interface in order for the terminal 100 to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language, such as machine language. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedure related control code necessary for the processor of the client to execute a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the client's processor to execute the functions at which location (address address) of the client's internal or external memory should be referenced. have. Also, if the processor of the client needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the client. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the client can access or various recording media on the client of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

Fourth, it is possible to obtain the positional relationship between the user's left hand and right hand through the vision sensor or the initial measurement position, the present invention can evaluate whether the positional relationship between the user's two hands is correct.

Fifth, it is possible to build a variety of item data to perform sign language learning necessary for the user's situation. In addition, it can be provided in combination with a variety of content, so that the user can enjoy learning sign language. In addition, through a sign language evaluation, a question suitable for a user's learning level may be suggested to provide a user with level-specific learning.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features thereof. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims

Extracting and providing specific item data by the terminal;

Extracting reference sign data corresponding to the item data;

Receiving sensing data from at least one body movement measuring device;

Generating input sign data by combining one or more pieces of the sensing data;

Calculating a comparison result by comparing the input sign language data with reference sign data; And

And performing an evaluation on the item data based on the comparison result.
The method of claim 1,

The reference sign data extraction step,

Sign language education method, characterized in that for extracting the reference sign data stored in matching with the item data stored in the database in the terminal.
The method of claim 1,

If the item data corresponds to a sentence,

The reference sign data extraction step,

Changing the order of the item data to a sign language order; And

And searching and matching sign language data corresponding to each word of the item data.
The method of claim 1,

When the body movement measuring device includes a glove-type measuring device and the attached measuring sensor device,

The sensing data receiving step,

Receiving sensing data of a finger movement and a wrist movement from the glove-type measuring device; And

And receiving sensing data from a measurement sensor device attached to each body unit.
The method of claim 4, wherein

The sensing data receiving step,

Requesting the user to perform a particular reference posture or reference move; And

And determining an initial position of each physical movement measuring device according to the reference posture or the standard movement.
The method of claim 5,

The input sign data generation step,

The sign language education method, characterized in that to calculate the positional relationship between the left hand and the right hand by tracking the movement of the body unit on the basis of the initial position.
The method of claim 4, wherein

When the body movement measuring device includes a vision sensor device,

Receiving an image acquired by the vision sensor device, wherein the image includes both hands of a user; And

Recognizing the positional relationship between the left and right hand in the image; Sign language education method further comprising.
The method of claim 1,

The evaluation step is,

If the item data is a word,

And calculating a correspondence rate between the reference sign language data and the input sign data of a word corresponding to the item data.
The method of claim 1,

The evaluation step is,

If the item data is a sentence,

Calculating, by the terminal, at least one reference word data of the reference sign data and at least one input word data of the input sign data, respectively, to calculate a matching rate;

Matching the input word data with the reference word data having the highest matching rate;

The word difference result is calculated by accumulating the distance difference between the input word data and the reference word data, wherein the distance difference is moved so that the specific word input data is placed in the same sentence position as the reference word data having the highest matching rate. A word count;

Calculating a word matching result based on a matching rate of the input word data matched with each of the reference word data; And

And calculating an evaluation score by reflecting the word matching result and the word matching result.
The method of claim 9,

And generating feedback data for the user based on the word matching result and the word matching result.
The method of claim 9,

Determining the difficulty of the next item data by reflecting the evaluation score; Sign language education method further comprising.
12. A sign language education program, coupled with a terminal, which is hardware, stored in a medium for carrying out the method of any one of claims 1 to 11.