KR20170116332A - Sign language recognition system and method - Google Patents

Abstract

A handwriting recognition system includes an acquisition unit for acquiring an EMG signal and an inertia signal of the user from a sensor measuring device worn on a user's arm, An extraction unit for extracting a muscle activity period from the muscle activity signal and extracting a motion duration of the arm region from the inertia signal, a first characteristic vector is calculated by performing signal processing on the muscle activity period, A search unit for searching for a signal corresponding to the integrated characteristic vector in the database based on an integrated characteristic vector obtained by integrating the first characteristic vector and the second characteristic vector; And an output unit for outputting a text corresponding to the searched signal.

Description

Technical Field [0001] The present invention relates to a sign recognition system and method,

The present invention relates to a sign language recognition system and method.

Sign language or finger language is a method of communication in which the hearing impaired and the speech impaired are represented by gestures or hand gestures on behalf of speech, Communication is performed through facial expressions or lip movements.

The conventional sign recognition system or the land recognition system has a problem that it takes a lot of time and is inconvenient to carry because it captures a hydration or an earthquake operation using a camera and analyzes the operation.

In recent years, there has been proposed a technique of recognizing hydration or hydration using hydration gloves. However, this technique is limited in that it can be worn for a long period of time due to reasons such as sweat on the hands, and there is a possibility that foreign matter When performing routine operations, there is a need to remove the hydration gloves.

The background technology of the present application is disclosed in Korean Patent Registration No. 10-1551424 (Registered on Feb. 25, 2015).

It is an object of the present invention to provide a sign language recognition system and method which are easy to carry without being restricted by the operation of everyday life.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a sign language recognition system and method capable of clearly identifying a sign language operation and a language sign language operation in a short period of time.

It is to be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to another aspect of the present invention, there is provided a handwriting recognition system including an acquisition unit for acquiring an EMG signal and an inertial signal of a user from a sensor measurement device worn on a user's arm, An extraction unit for extracting a muscle activity segment from the EMG signal and extracting a motion segment of the arm region from the inertia signal to sense a hydration operation taken by the user, Calculating a second characteristic vector by performing a signal processing on the motion section, calculating a second characteristic vector based on the integrated characteristic vector obtained by integrating the first characteristic vector and the second characteristic vector, A search unit for searching for a signal corresponding to the feature vector, and an output unit for outputting a text corresponding to the searched signal And a force part.

The acquiring unit may include a first receiving unit that receives an EMG signal corresponding to an operation of waving the user's wrist through a plurality of electrode channels included in the sensor measuring device, A channel identification unit for identifying an electrode channel having a maximum effective output value among the plurality of electrode channels based on the position of the electrode channel detected by the sensor measuring unit, And an alignment unit for rearranging the plurality of electrode channels.

The acquisition unit may include a second receiving unit that receives the inertial signal through an inertial measurement unit included in the sensor measuring device, a second receiving unit that receives the inertial measurement initial value of the inertial measurement unit and the inertial measurement unit of the inertial measurement unit using the inertial signal, A calculation unit for calculating an orientation value of the inertia measurement unit according to real-time reception, a calculation unit for calculating a roll angle of the inertia measurement unit based on the reference vector determined based on the orientation initial value and a motion vector determined based on the orientation value, And an arithmetic unit for calculating the yaw angle.

The extraction unit extracts the muscle activity period by applying a Tee-Kaiser Energy Operator (TKEO) technique to the EMG signals received from each of the plurality of electrode channels included in the sensor measurement device, The motion interval may be extracted by applying a Teeter-Kaiser Energy Operator (TKEO) technique to the inertia signal received from the included inertial measurement unit.

Also, the extracting unit may extract, as the muscle activity period, an interval that is equal to or greater than a preset muscle activity threshold value in the EMG signal, and extract the interval that is equal to or greater than a predetermined motion threshold value in the inertia signal as the motion interval.

The calculating unit may calculate the first characteristic vector by calculating an effective output value of the electromyogram signal for each of the plurality of electrode channels included in the sensor measuring device based on the muscle activity period, The second feature vector can be calculated by applying a high-pass filter.

In addition, the first feature vector and the second feature vector may be resampled by normalizing the time data.

In addition, the search unit may perform the search using a neural network formed through learning of a specific hydration operation.

The sensor measuring device may further comprise an armband to be worn to surround the arm portion, a plurality of electrodes spaced apart from each other along the inner circumference of the armband so as to face the arm portion, and an inertia The inertial measurement unit may include a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor.

Meanwhile, a sign language recognition method according to an embodiment of the present invention includes the steps of acquiring an EMG signal and an inertia signal of the user from a sensor measurement device worn on a user's arm, Extracting a muscle activity section from the EMG signal, extracting a motion section of the arm region from the inertial signal, calculating a first characteristic vector by performing signal processing on the muscle activity section, Searching for a signal corresponding to the integrated characteristic vector in a database based on an integrated characteristic vector obtained by integrating the first characteristic vector and the second characteristic vector, And outputting a text corresponding to the signal.

The acquiring step may include receiving an EMG signal according to an operation of waving the wrist of the user through a plurality of electrode channels included in the sensor measuring device and receiving the EMG signal received from each of the plurality of electrode channels A plurality of electrode channels, each electrode channel having a maximum effective output value among the plurality of electrode channels, based on the position of the electrode channel in the sensor measuring device, Lt; / RTI >

The acquisition step may include receiving the inertial signal through an inertial measurement unit included in the sensor measurement device and using the inertial signal to calculate an orientation initial value of the inertial measurement unit and a real- A pitch angle and a yaw angle of the inertia measurement unit using a motion vector determined based on the reference vector determined based on the orientation initial value and the orientation value, Can be calculated.

The extracting step may include extracting the muscle activity section by applying a teeter-Kaiser energy operator (TKEO) technique to the EMG signal received from each of the plurality of electrode channels included in the sensor measurement device, The motion interval can be extracted by applying a Teeter-Kaiser Energy Operator (TKEO) technique to the inertia signal received from the inertial measurement unit included in the apparatus.

Also, the extracting step may extract, as the muscle activity section, a section that is equal to or greater than a muscle activity threshold value set in advance in the EMG signal, and extract a section that is equal to or greater than a predetermined motion threshold value in the inertia signal as the motion section.

The calculating step calculates the first characteristic vector by calculating an effective output value of the electromyogram signal for each of the plurality of electrode channels included in the sensor measuring device on the basis of the muscle active section, The second feature vector can be calculated by applying a high-pass filter.

In addition, the first feature vector and the second feature vector may be resampled by normalizing the time data.

In addition, the searching may be performed using a neural network formed through learning of a specific hydration operation.

The sensor measuring device may further comprise an armband to be worn to surround the arm portion, a plurality of electrodes spaced apart from each other along the inner circumference of the armband so as to face the arm portion, and an inertia The inertial measurement unit may include a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor.

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-mentioned problem solving means of the present invention, the user's EMG signal and inertial signal are acquired from the sensor measuring device worn on the arm portion of the user, and the characteristic vector of the hydration operation taken by the user is calculated based on the EMG signal and the inertia signal And outputting a text corresponding to the calculated characteristic vector in the database, thereby making it possible to recognize the intention through the sign language operation more quickly and accurately.

According to the above-mentioned problem solving means of the present invention, it is possible to measure an electromyogram signal and an inertia signal according to a user's hydration operation through a wearable sensor measuring device on a user's arm, It is possible to provide a sign language recognition system and method which is easy to carry without being restricted by the operation of daily life.

According to the above-described task resolution means of the present invention, since the user's handwriting operation is recognized by considering the electromyogram signal acquired from the electromyogram sensor and the three-dimensional angle of the arm portion of the user acquired through the inertia measurement unit, There is an effect that can be recognized more accurately.

1 is a diagram schematically showing the entire configuration of a sign language recognition system according to an embodiment of the present invention.
2 is a diagram schematically showing the configuration of an acquiring unit in a sign language recognition system according to an embodiment of the present invention.
3 is a diagram illustrating a sensor measurement apparatus used in a sign recognition system according to an embodiment of the present invention.
4 is a diagram illustrating an example of a TKEO technique used in a sign recognition system according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating an example of detection of a muscle activity interval and a motion interval in the sign recognition system according to an embodiment of the present invention.
FIG. 6 is a graph illustrating an integrated characteristic vector calculated through the sign language recognition system according to an embodiment of the present invention.
7 is a diagram illustrating an example of a neural network used in the sign language recognition system according to an embodiment of the present invention.
8 is a diagram illustrating an example of detecting a signal corresponding to an integrated characteristic vector in the sign recognition system according to an embodiment of the present invention.
9 is a schematic operation flowchart of a sign language recognition method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when an element is referred to as being "connected" to another element, it is intended to be understood that it is not only "directly connected" but also "electrically connected" or "indirectly connected" "Is included.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

The present invention relates to a handwriting recognition system and method for recognizing a handwriting operation and a handwriting operation using an EMG signal and an inertial signal.

FIG. 1 is a view schematically showing the entire configuration of a sign language recognition system according to an embodiment of the present invention, and FIG. 2 is a schematic view of the configuration of an acquisition unit in a sign language recognition system according to an embodiment of the present invention.

1, the sign language recognition system 100 according to an embodiment of the present invention includes an acquisition unit 110, an extraction unit 120, a calculation unit 130, a search unit 140, and an output unit 150 .

The acquiring unit 110 may acquire the user's EMG signal and inertial signal from the sensor measuring device worn on the user's arm. The sensor measuring instrument used for obtaining the EMG signal and the inertial signal in the present application can be more easily understood with reference to Fig.

3 is a diagram illustrating a sensor measurement apparatus used in a sign recognition system according to an embodiment of the present invention.

Referring to FIG. 3, the sensor measuring instrument 10 according to an embodiment of the present invention may be worn on the arm portion of the user. In particular, the sensor measuring instrument 10 may be worn on the upper arm portion, which is a part between the elbow and the shoulder, in addition to the forearm portion between the wrist and the elbow.

The sensor measuring device 10 includes an armband 11, a plurality of electrodes (e.g., a first electrode 1, a second electrode 2, a third electrode 3, ...) and an inertial measurement unit (IMU, Intertial Measurement Unit (12). The armband 11 may be a band worn to surround the arm portion of the user. The armband 11 may be a material that can expand or contract depending on the thickness of the body part of the user to which the sensor measuring device 10 is worn. The plurality of electrodes 1, 2, 3, ... may be disposed at intervals along the inner circumference of the armband 11 so as to face the arms of the user. The plurality of electrodes 1, 2, 3, ... may be electromyogram electrodes. The inertial measurement unit 12 may be formed in one area of the sensor measuring instrument 10. [ The inertial measurement unit 12 may include a triaxial accelerometer, a triaxial gyroscope, and a triaxial geomagnetic sensor.

In addition, the sensor measuring instrument 10 may include a control unit (not shown). The sensor measuring instrument 10 can measure the user's electromyogram signal through the plurality of electrodes 1, 2, 3, ... based on the control signal of the controller. Further, the sensor measuring instrument 10 can acquire the inertia signal through the inertia measurement unit 12 based on the control signal of the control unit. The sign language recognition system 100 according to an embodiment of the present invention can clearly identify the sign language operation that the user has taken based on the measured EMG signal and the measured inertia signal.

In addition, the control unit transmits the EMG signal measured through the plurality of electrodes and the inertial signal measured through the inertial measurement unit 12 to the sign recognition system 100 (100) via wireless communication such as Bluetooth, NFC (near field communication) ). Accordingly, the acquisition unit 110 of the sign language recognition system 100 can acquire the user's EMG signal and the inertial signal from the sensor measurement device 10. [

Referring to FIG. 2, the acquiring unit 110 may include a first acquiring unit 110a for calibrating an EMG signal and a second acquiring unit 110b for calibrating an inertial signal. The first obtaining unit 110a may include a first receiving unit 111, a channel identifying unit 112 and an aligning unit 113. The second obtaining unit 110b may include a second receiving unit 114, (115), and an operation unit (116).

Calibration is performed by setting the characteristics (or scale) of the subject (i.e., the user) to the EMG signal measured through the plurality of electrodes 1, 2, 3, ... and the inertial signal measured through the inertial measurement unit 12. [ And the standard is adjusted to a certain standard. Accordingly, the sign language recognition system 100 according to an embodiment of the present invention can more accurately analyze the electromyogram signal and the inertia signal measured through the sensor measuring device 10 in consideration of the characteristics of the user.

The sign language recognition system 100 according to an embodiment of the present invention can recognize the value of the electromyogram signal measured from the sensor measuring instrument 10 through the first receiving unit 111, the channel identifying unit 112 and the aligning unit 113 Calibration can be done.

The first receiving unit 111 can receive an EMG signal according to a user's wrist swinging operation through a plurality of electrode channels included in the sensor measuring device 10. [ The plurality of electrode channels means channels corresponding to the plurality of electrodes 1, 2, 3, ..., respectively.

The channel identification unit 112 can identify an electrode channel having a maximum effective power (RMS) value among a plurality of electrode channels, based on the electromyogram signal received from each of the plurality of electrode channels.

The channel identification unit 112 can identify the position of the electrode channel where the maximum effective output value among the plurality of electrode channels appears by comparing the EMG signals received from each of the plurality of electrode channels.

The position of the electrode channel having the maximum effective output value may be the position of a wrist extensor bundle. Accordingly, the channel identifying unit 112 can detect the position of the wrist extensor bundle by identifying the position of the electrode channel where the maximum effective output value appears.

The position of the electrode channel identified in the channel identification unit 112 may be stored in a database (not shown).

The aligning unit 113 may rearrange a plurality of electrode channels included in the sensor measuring device 10 in consideration of the position of the electrode channel identified through the channel identifying unit 112 for the measurement of a constant EMG signal . The user wears the sensor measuring instrument 10 and then performs the initial calibration of the electrode channel through this reordering so that the acquiring unit 110 acquires the electromyogram signal corresponding to the wrist extensor herniation associated with the wrist swinging operation with high accuracy Can be set.

The sign language recognition system 100 can also calibrate the value of the inertial signal measured from the sensor measuring instrument 10 through the second receiving unit 114, the calculating unit 115 and the calculating unit 116 .

The second receiving unit 114 can receive the inertia signal according to the user's movement through the inertial measurement unit 12 included in the sensor measuring device 10. [

), 피치(Pitch) 회전에 대응하는 피치 각도(쎄타, θ) 및 요(Yaw) 회전에 대응하는 요 각도(프사이, ψ)를 포함할 수 있다.The calculation unit 115 may calculate the initial value of the orientation of the inertial measurement unit 12 using the inertial signal measured through the inertial measurement unit 12. [ The orientation initial values of the inertia measurement unit 12 include roll angles (pi,

), A pitch angle (theta, [theta]) corresponding to the pitch rotation, and a yaw angle (psi, [psi]) corresponding to the yaw rotation.

Hereinafter, an example of calculating the initial orientation value of the inertia measurement unit 12 through the equations (1) to (8) will be described.

The calculation unit 115 calculates the inertia measurement unit 12 using the acceleration signal of the x-axis of the inertia measurement unit 12, the y-axis acceleration signal of the inertia measurement unit 12, The initial value of the roll angle of the inertia measurement unit 12 and the initial value of the pitch angle of the inertia measurement unit 12 can be determined. The following equations (1) and (2) show an example in which the initial value of the roll angle and the initial value of the pitch angle are determined by the calculation unit 115 as an example.

)와 피치 각도인 쎄타(θ)를 구하기 위해, 관성측정유닛(12)의 x축의 가속도 신호인 a_x, 관성측정유닛(12)의 y축의 가속도 신호인 a_y 및 관성측정유닛(12)의 z축의 가속도 신호인 a_z와 중력 가속도 벡터인 g를 이용할 수 있다. 이때,

는 변환행렬을 의미할 수 있으며, 이러한

는 수학식 2와 같이 나타낼 수 있다.Referring to Equation (1), the calculation unit 115 calculates a roll angle pie

Axis acceleration signal a _x of the inertia measurement unit 12 and the y-axis acceleration signal a _y of the inertia measurement unit 12 and the acceleration signal a _y of the inertia measurement unit 12 of the inertia measurement unit 12, the acceleration signal a _z of the z axis and the acceleration g of the gravity acceleration vector g can be used. At this time,

May refer to a transformation matrix,

Can be expressed by Equation (2).

는 기준 좌표계에서 동체 좌표계로의 변환을 나타내는 항법 좌표계를 의미할 수 있다. 수학식 2에 나타난 바와 같이, 변환행렬

는 롤 각도의 변환, 피치 각도의 변환 및 요 각도의 변환을 요소들로 가질 수 있으며, 롤 각도의 변환, 피치 각도의 변환 및 요 각도 각각은

,

,

로 표현될 수 있다.Referring to Equation (2), the transformation matrix

May refer to a navigation coordinate system that represents the transformation from the reference coordinate system to the body coordinate system. As shown in Equation (2), the transformation matrix

Can convert the roll angle, the pitch angle, and the yaw angle, respectively, and the roll angle conversion, the pitch angle conversion, and the yaw angle, respectively,

,

,

. ≪ / RTI >

), 피치 각도인 쎄타(θ)의 값을 계산할 수 있다. 계산부(115)는 계산한 롤 각도인 파이(

), 피치 각도인 쎄타(θ) 각각을 롤 각도의 초기값 및 피치 각도의 초기값으로 결정할 수 있다. 한편, 요 각도는 프사이(ψ)로 표현될 수 있다.Referring to equations (1) and (2), the calculation unit 115 calculates a value of 1 (g: gravitational acceleration) as the z axis of the x axis, the y axis and the z axis, And determining the remaining two axes, x-axis and y-axis, to be 0,

), And the pitch angle, theta ([theta]). The calculation unit 115 calculates the roll angle pie

) And the pitch angle (?) Can be determined as an initial value of the roll angle and an initial value of the pitch angle, respectively. On the other hand, the yaw angle can be expressed by the psi ().

The calculation unit 231 calculates the inertia measurement unit 12 using the geomagnetic signals of the x-axis of the inertia measurement unit 12, the y-axis earth signals of the inertia measurement unit 12 and the z- The initial value of the yaw angle of the yaw axis 12 can be determined. Hereinafter, an example of determining the initial value of the yaw angle by the calculation unit 115 through Equations (3) to (8) will be described.

및 지구의 지자계 벡터인 (m₁, m₂, m₃)를 이용하여 요 각도인 프사이(ψ)의 초기값을 계산할 수 있다.Referring to Equation 3, the calculation section 231 first prophet x axis of the sensor 110, the signals in the m _x, the first prophet of the y-axis of the sensor 110, the signals in the m _y and the first sensor (110 ) m _z, the transformation matrix prophet z axis based on signals

(M ₁ , m ₂ , m ₃ ) of the earth's earth coordinate system can be used to calculate the initial value of the yaw angle ψ.

(

)은 수학식 4와 같이 롤 각도와 피치 각도와 연관된 C₁과 요 각도와 연관된 C₂로 표현될 수 있으며, 수학식 1은 수학식 5 및 수학식 6으로 변환될 수 있다.Meanwhile,

(

Can be expressed as C ₁ associated with the roll angle and pitch angle and C ₂ associated with the yaw angle, as shown in equation (4), and equation (1) can be transformed into equation (5) and equation (6).

) 및 피치 각도인 쎄타(θ) 각각이 0일 때의 지자계 신호인 m_x, m_y 및 m_z와 지구의 지자계 벡터의 값인 m₁, m₂ 및 m₃를 대입하여, 요 각도의 초기값을 구할 수 있다.Substituting C ₁ and C ₂ shown in Equation (4) into Equation (6) yields Equation (7) and Equation (8). The calculation unit 115 calculates the roll angle Pa

M ₁ , m _2, and m ₃ , which are the values of the earth's magnetic field signals m _x , m _y, and m _z when the pitch angle θ and theta angle of the pitch angle are 0, Value can be obtained.

) 초기값, 피치 각도(쎄타, θ) 초기값 및 요 각도(프사이, ψ) 초기값을 계산할 수 있다. 계산부(115)는 계산된 오퍼레이션 초기값에 기초하여 기준 벡터(Vector ₀ )를 결정할 수 있다. 기준 벡터를 결정함으로써 관성 신호에 대한 초기 캘리브레이션 수행이 완료될 수 있다.As described above, the calculation unit 115 calculates the initial orientation of the inertia measurement unit 12, that is, the roll angle of the inertia measurement unit 12 (pi,

) Initial value, pitch angle (theta, θ) initial value and yaw angle (psi, ψ) initial value can be calculated. The calculation unit 115 can determine the reference vector (Vector ₀ ) based on the calculated operation initial value. The initial calibration for the inertial signal can be completed by determining the reference vector.

Thereafter, the calculation unit 115 can calculate the orientation value of the inertia measurement unit 12 according to the real-time reception of the inertia signal as it receives the inertia signal measured through the inertia measurement unit 12 in real time. That is, the calculation unit 115 may calculate the orientation value of the inertia measurement unit 12 in real time through Equations (1) to (8) after calculating the operation initial value. Calculation section 115 may be using the orientation values computed in real time to determine the motion vector (Vector _move).

The calculation unit 116 calculates the position of the roll of the inertial measurement unit 12 using the motion vector (Vector _move ) determined based on the reference vector (Vector ₀ ) determined based on the orientation initial value of the inertia measurement unit 12 and the orientation value The angle, pitch angle and yaw angle can be calculated. Here is a closer look.

First, the operation unit 116 may calculate a quaternion element when rotating from a reference vector (Vector ₀ ) to a motion vector (Vector _move ). The quaternion element can be expressed as Equation (9).

In order to calculate the quaternion element, the operation unit 116 may apply a Half-Way Quaternion Solution that computes the quaternion element using the inner product and the outer product between the two vectors.

Accordingly, the reference vector (Vector ₀₎ and the motion vector (Vector _move) Q1 of angular components between, the product of the inner product and the vector size as shown in Equation 10, the reference vector (Vector ₀₎ and the motion vector (Vector _move) Lt; / RTI >

Q2, Q3 and Q4 which are the x, y and z components of the rotation axis can be derived out of the x, y and z component vectors between the reference vector (Vector ₀ ) and the movement vector (Vector _move ) Can be expressed as:

Then, the computing unit 116 can calculate the roll angle, the pitch angle, and the yaw angle of the inertia measurement unit 12 based on the quaternion-Euler transformation formula, which can be expressed as shown in Equation (12).

The extraction unit 120 may extract a muscle activity period from the electromyogram signal acquired by the acquisition unit 110 to sense a hydration operation taken by the user. In addition, the extraction unit 120 may extract a motion interval from the inertia signal acquired by the acquisition unit 110. First, an extraction example of the muscle active section will be described.

The extraction unit 120 may apply a band-pass filter to the electromyogram signal acquired by the acquisition unit 110 before extracting the muscle activity period. For example, the extraction unit 120 may apply a band pass filter of 10 to 450 Hz to the acquired electromyogram signal. The extraction unit 120 may apply an analog-to-digital converter (ADC) to the acquired EMG signal.

The extraction unit 120 may extract a muscle activity period by applying a Teeter-Kaiser Energy Operator (TKEO) technique to the EMG signals received from each of the plurality of electrode channels. In addition, the extracting unit 120 may extract, as a muscle activity period, an interval that is equal to or greater than a preset muscle activity threshold value in the EMG signal acquired by the acquisition unit 110. [

The Teaker-Kaiser Energy Operator (TKEO) technique is a signal processing technique that extracts muscle activity in very small movements such as finger movements. It uses a muscle activity of finger movements with a low signal-to-noise ratio (SNR) Can be detected. The TKEO technique is a well known technique to those skilled in the art. Therefore, rather than describing the TKEO technique itself, the TKEO technique is applied to a sign language recognition system according to an embodiment of the present invention I will explain.

4 is a diagram illustrating an example of a TKEO technique used in a sign recognition system according to an embodiment of the present invention.

Referring to FIG. 4, for example, FIG. 4A shows a graph of a signal to which the TKEO technique is not applied, and FIG. 4B shows a graph of a signal to which the TKEO technique is applied to the signal of FIG. . A graph in which a low pass filter of 50 Hz is applied is shown in Fig. 4 (a ') and a graph in which a low pass filter of 50 Hz is applied in Fig. 4 (b) 4 (b '). In the case of FIG. 4 (b '), it can be seen that the probability of false detection in the active section is reduced by significantly increasing the signal-to-noise ratio (SNR) compared to FIG. 4 (a').

The extraction unit 120 may apply the TKEO technique to each of the EMG signals received through the plurality of electrode channels in order to extract the muscle activity period.

When the TKEO technique is applied to the electromyogram signal received from the sensor measuring instrument 10, it can be expressed as Equation (13).

The extraction unit 120 may synthesize data (electromyogram signals) of all the channels (i.e., a plurality of electrode channels) to which the TKEO technique is applied, and may be expressed as Equation (14).

는 채널1의 근전도 신호(EMG)를 나타내고,

는 채널2의 근전도 신호를 나타내며,

는 채널8의 근전도 신호를 나타낸다.here,

Represents the EMG signal EMG of the channel 1,

Represents an EMG signal of channel 2,

Represents an EMG signal of channel 8.

Then, the extracting unit 120 may calculate an effective output (RMS) value for the combined data in which the data of all the channels are combined. Effective output value for the combined data (U _{_ RMS EMG)} may be expressed as Equation (15).

In this case, N represents a window width (window width), U_emg (n) denotes the composite data of the EMG signal is TKEO formula applied, U _{RMS _ EMG} represents the RMS data of the synthesized EMG signal (that is, the composite data) .

The extraction unit 120 may perform a rectification process, which is a process of obtaining the absolute value of the EMG signal after calculating the effective output value for the combined data.

The extracting unit 120 extracts a linear envelope (EM) signal obtained by simplifying the EMG signal by performing a band-pass filter, a low-pass filter, a TKEO technique, rectification, Linear Envelope) signal. Then, the extracting unit 120 can extract the muscle active section based on the linear envelope signal. An example of extracting the muscle active section can be more easily understood with reference to FIG.

FIG. 5 is a diagram illustrating an example of detection of a muscle activity interval and a motion interval in the sign recognition system according to an embodiment of the present invention. Fig. 5A shows an example of detection of a muscle activity section, and Fig. 5B shows an example of detection of a motion section. Hereinafter, only a detection example of a muscle active section will be described with reference to FIG. 5 (a), and FIG. 5 (b) will be described later.

First, a threshold value for detecting a muscle active section may be preset by user input. The threshold value can be defined as 'Baseline average + J * standard deviation'. In this case, the baseline means an electromyogram signal measured when the user is not giving a force, and j means a constant value.

The threshold value is a measure for determining whether or not the muscles of the subject are muscular active. If the measured value of the electromyogram signal is greater than or equal to the threshold value, the threshold value is determined to be the muscle active state. , It can be judged as a muscle active off state.

In Fig. 5 (a), a point indicates a point where muscle activity is ON, and a point b indicates a point where muscle activity is turned off. Therefore, the section between point a and point b can be said to be the muscle active section.

The extracting unit 120 sets a point at which the EMG signal obtained by the acquiring unit 110 rises above the threshold value to a muscle active ON point (for example, a point), and falls to a point below the threshold value By setting it to the muscle active OFF point (for example, the b point), the muscle activity period can be extracted.

The extraction unit 120 may determine that the user has performed the hydration operation or the gazing operation when the muscle activity interval is detected in the EMG signal acquired by the acquisition unit 110 so as to stop the measurement of the EMG signal . The extraction unit 120 may deactivate the acquisition unit 110 and activate the calculation unit 130 when a muscle active period is detected. The extraction unit 120 may deactivate the calculation unit 130 and activate the acquisition unit 110 when the muscle active period is not detected. In addition, the EMG signal measurement from the sensor measuring device 10 can be stopped by user input.

Hereinafter, an example of extracting a motion interval based on an inertia signal will be described.

The extraction unit 120 may apply a band-pass filter to the inertial signal acquired by the acquisition unit 110 before extracting the motion interval from the inertial signal. The extraction unit 120 can extract the acceleration, the angular velocity, and the signal spectrum of the geomagnetic sensor included in the inertial measurement unit 12.

The extraction unit 120 may calculate the signal vector magnitude (SVM) of the acceleration signal based on the inertia signal acquired through the inertia measurement unit 12, Can be expressed as.

Then, the extraction unit 120 may apply the TKEO equation to the SVM signal of the acceleration, which can be expressed by Equation (17).

Then, the extraction unit 120 can calculate the effective output (RMS) value of Equation (17), which can be expressed as Equation (18).

In this case, N represents a window width (window width), Ψ_acc (n ) denotes the acceleration signal is applied TKEO formula, U _{_ RMS acc} represents the RMS data of the acceleration signal TKEO formula is applied.

Then, the extracting unit 120 can extract the measured motion interval as the user takes the hydration operation in the inertia signal, based on the RMS data of the acceleration signal to which the TKEO formula is applied. An example of the extraction of the motion interval can be more easily understood with reference to FIG. 5 (b).

5B, the extractor 120 determines that the time when the inertia signal obtained from the sensor measuring instrument 10 is rising above a predetermined threshold value is a motion on state, and when the time falls below the threshold value Can be judged as a motion off state.

In Fig. 5 (a), a point represents a point at which motion is ON, and a point b represents a point at which motion is OFF. Therefore, the interval between points a and b can be called a motion interval.

The extracting unit 120 sets a point at which the inertial signal obtained by the obtaining unit 110 goes up above the threshold value as a motion ON point (for example, a point), moves a point lower than the threshold value OFF point (for example, point b), motion cycles can be extracted.

The extraction unit 120 may determine that the user has performed a hydration operation or an attenuation operation when the motion interval is detected in the inertia signal acquired by the acquisition unit 110, and stop the measurement of the inertia signal. The extraction unit 120 may deactivate the acquisition unit 110 and activate the calculation unit 130 when a motion interval is detected. The extraction unit 120 may deactivate the calculation unit 130 and activate the acquisition unit 110 when no motion interval is detected. In addition, inertial signal measurement from the sensor measuring instrument 10 may be interrupted by user input.

Through the above process, the extraction unit 120 extracts, as a muscle activity period, an interval that is equal to or greater than a preset muscle activity threshold value in the EMG signal, and extracts an interval that is equal to or greater than a preset motion threshold value in the inertia signal as a motion interval.

Next, the calculating unit 130 calculates the first characteristic vector of the hydration operation taken by the user by performing signal processing on each of the muscle active period and the motion period extracted by the extracting unit 120, The second characteristic vector of the hydration operation taken by the user can be calculated. First, an example of calculating the first characteristic vector will be described below.

The calculating unit 130 calculates the effective output value (RMS) of the electromyogram signal for each of the plurality of electrode channels included in the sensor measuring instrument 10, based on the electromyogram signal included in the muscle activity period, Can be calculated. At this time, the calculating unit 130 may calculate the first characteristic vector of the hydration operation taken by the user in consideration of the position of the electrode channel identified by the channel identifying unit 112. [ In addition, the calculator 130 may remove the data of the remaining interval excluding the muscle activity period from the EMG signal.

The calculating unit 130 may calculate the effective output value (FRMS _C ) of the electromyogram signal for each channel in the muscle activity period through the following equation (19).

At this time, C represents the channel number of the electrode and τ represents the muscle active period. For example, the channel number of the first electrode 1 may be 1, and the channel number of the second electrode 2 may be 2.

The calculation unit 130 may calculate the first characteristic vector by normalizing the time data based on the effective output value FRMS _C calculated through the equation (19). The first feature vector may be resampled by normalizing the time data to facilitate comparison with feature vectors stored in a database (not shown). Hereinafter, an example of calculation of the second characteristic vector will be described.

The calculating unit 130 may remove the data of the remaining interval excluding the motion interval from the inertia signal. For example, the calculating unit 130 may remove all rolls, pitches, and yaw data except for the period in which the motion is detected through the acceleration signal.

Also, the calculator 130 may apply an Euler high-pass filter to the extracted motion section. The calculation unit 130 can apply a high-pass filter of 0.1 Hz, for example, and can thereby calculate the second characteristic vector by fixing the offset of roll, pitch, and yaw data to zero. At this time, the first characteristic vector may be resampled by normalizing the time data by the calculation unit 130 so that it can be easily compared with the characteristic vectors stored in the database (not shown).

Further, the calculating unit 130 may integrate the first characteristic vector calculated based on the EMG signal and the second characteristic vector calculated based on the inertia signal through Equation (20) to calculate an integrated characteristic vector have.

FRMS ₁ represents the effective output value of the electromyogram signal obtained through the channel of the first electrode ₁ in the muscle active section, FRMS ₂ represents the muscle output signal of the muscular activity Represents the effective output value of the electromyogram signal obtained through the channel of the second electrode 2 in the section. In addition, the roll, pitch, and yaw represent the effective output value of the inertia signal (i.e., IMU signal) obtained through the inertia measurement unit 12.

FIG. 6 is a graph showing an integrated characteristic vector calculated through the sign language recognition system according to an embodiment of the present invention. For example, the integrated characteristic vector of the hydration operation taken by the user, 6, respectively.

The searching unit 140 searches for a signal corresponding to the integrated characteristic vector in the database (not shown) based on the integrated characteristic vector obtained by combining the first characteristic vector and the second characteristic vector calculated by the calculating unit 130 .

At this time, the search unit 140 can perform a search using a neural network formed through learning of a specific sign language operation or an intelligence operation. An example of a neural network is shown in Fig.

7 is a diagram illustrating an example of a neural network used in the sign language recognition system according to an embodiment of the present invention.

Referring to FIG. 7, the searching unit 140 can search for a signal corresponding to the integrated characteristic vector calculated by the calculating unit 130 in a database (not shown) more quickly and accurately through a pattern recognition method based on a neural network have. For this purpose, the search unit 140 can determine the parameters (W, bias) of the neural network that maximizes the pattern classification probability through learning of a specific hydration operation or an artificial operation.

The search unit 140 can search for and extract a signal having the highest similarity to the integrated characteristic vector among the signals included in the database. This can be more easily understood with reference to Fig.

8 is a diagram illustrating an example of detecting a signal corresponding to an integrated characteristic vector in the sign recognition system according to an embodiment of the present invention.

Referring to FIG. 8, FIG. 8 (a) shows a graph 80 of the integrated characteristic vector calculated through the calculation unit 130.

8 (b) shows an example of signals stored in the database, which will be described in more detail as follows. The waveform graph corresponding to the integrated characteristic vector of the EMG signal and the inertia signal corresponding to each text may be stored in the database by text (e.g., words, alphabets, letters, numbers, consonants, vowels, etc.). For example, the graph corresponding to the integrated characteristic vector of the hydration operation representing the word " Hello " may be the same as the first graph 81. The graph corresponding to the integrated feature vector of the hydration operation representing the word " Bedroom " may be the same as the second graph 82. The graph corresponding to the integrated characteristic vector of the hydration operation representing the word " Want " may be the same as the third graph 83. The graph corresponding to the integrated characteristic vector of the hydration operation representing the word 'Lunch' may be the same as the fourth graph 84. The graph corresponding to the integrated characteristic vector of the hydration operation representing the word " Music " may be the same as the fifth graph 85. The graph corresponding to the integrated characteristic vector of the hydration operation representing the word " Airplane " may be the same as the sixth graph 86.

The search unit 140 can search for the signal corresponding to the graph 80 shown in FIG. 8 (a) in the data (database) shown in FIG. 8 (b). The search unit 140 can extract the second graph 82 in the database as a search result of the signal corresponding to the integrated characteristic vector.

Then, the output unit 150 may output the text corresponding to the search result in the search unit 140 (i.e., the word 'Bedroom').

The output unit 150 may output the text corresponding to the signal retrieved by the retrieval unit 140 to a display screen or a speaker.

The sign language recognition system 100 according to one embodiment of the present invention can be performed in a portable terminal, a smart phone, a personal digital assistant (PDA), a tablet, a notebook PC, a desktop PC, and the like, but is not limited thereto.

The output unit 150 may output the text corresponding to the signal retrieved from the retrieval unit 140 to a display screen of a user terminal such as a portable terminal, a smart phone, a desktop PC, or the like.

Hereinafter, the operation flow of the present invention will be briefly described based on the details described above.

9 is a schematic operation flowchart of a sign language recognition method according to an embodiment of the present invention. The sign language recognition method shown in FIG. 9 can be performed by the sign language recognition system 100 described above with reference to FIGS. 1 to 8. FIG. Therefore, the contents described with respect to the sign language recognition system 100 through Figs. 1 to 8 can be applied to Fig. 9 even if omitted below.

Referring to FIG. 9, in step S910, the user's electromyogram signal and inertial signal may be obtained from the sensor measuring device 10 worn on the user's arm through the acquiring unit 110. FIG.

In step S910, the obtaining unit 110 can receive the EMG signal according to the user's wrist swinging operation through the plurality of electrode channels included in the sensor measuring instrument 10. [ Also, the acquiring unit 110 can identify the electrode channel having the maximum effective output value among the plurality of electrode channels, based on the electromyogram signal received from each of the plurality of electrode channels. The acquiring unit 110 can also reorder the plurality of electrode channels included in the sensor measuring device 10 in consideration of the position of the electrode channel identified in the sensor measuring device 10 for constant sensor metering .

Further, in step S910, the obtaining unit 110 may receive the inertial signal through the inertial measurement unit 12 included in the sensor measuring instrument 10. [ The acquiring unit 110 may calculate an orientation initial value of the inertial measurement unit 12 and an orientation value of the inertial measurement unit 12 according to real-time reception of the inertia signal using the received inertia signal. In addition, the acquiring unit 110 may calculate the roll angle, the pitch angle, and the yaw angle of the inertia measurement unit 12 using the motion vector determined based on the reference vector and the orientation value determined based on the orientation initial value.

Next, in step S920, in order to sense a hydration operation taken by the user through the extraction unit 120, a muscle activity section may be extracted from the EMG signal and a motion section may be extracted from the inertia signal.

In step S920, the extracting unit 120 extracts a muscle activity interval by applying a Tee-Kaiser Energy Operator (TKEO) technique to the EMG signals received from each of the plurality of electrode channels, and calculates an inertia measurement A motion interval can be extracted by applying a Tee-Kaiser Energy Operator (TKEO) technique to the inertial signal received from the unit 12.

In step S920, the extracting unit 120 extracts, as a muscle activity section, a section having a muscle activity threshold value or more set in advance in the EMG signal, and extracts a section having a motion threshold value equal to or greater than a preset motion threshold value as a motion section.

Next, in step S930, signal processing is performed on the muscle activity interval extracted in step S920 via the calculation unit 130 to calculate a first feature vector for a hydration operation taken by the user, and signal processing is performed on the motion interval The second characteristic vector for the hydration operation taken by the user can be calculated.

In step S930, the calculating unit 130 calculates the first characteristic vector by calculating the effective output value of the electromyogram signal for each of the plurality of electrode channels included in the sensor measuring instrument 10 based on the muscle active period, It is possible to calculate the second feature vector by applying the high pass filter based on the interval. At this time, the first characteristic vector and the second characteristic vector may be resampled by normalizing the time data. Further, the calculating unit 130 may calculate the integrated characteristic vector for the hydration operation taken by the user based on the first characteristic vector and the second characteristic vector.

Next, in step S940, a signal corresponding to the integrated characteristic vector may be searched in the database based on the integrated characteristic vector obtained by combining the first characteristic vector and the second characteristic vector through the search unit 140. [

In step S940, the search unit 140 may perform a search using a neural network formed through learning of a specific hydration operation.

Next, in step S950, the text corresponding to the signal retrieved in step S940 may be output through the output unit 150. [

In the above description, steps S910 to S950 may be further divided into further steps or combined into fewer steps, according to embodiments of the present disclosure. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed.

The sign language recognition method according to an exemplary embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and configured for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The above-described sign language recognition method may also be implemented in the form of a computer program or an application executed by a computer stored in a recording medium.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included within the scope of the present invention.

100: Sign Language Recognition System
110: Acquiring unit 120:
130: Calculator 140: Search unit
150:

Claims (19)

An acquiring unit for acquiring an EMG signal and an inertia signal of the user from a sensor measuring device worn on a user's arm;
An extraction unit for extracting a muscle activity segment from the EMG signal and extracting a motion segment of the arm region from the inertial signal to sense a hydration operation taken by the user;
A calculating unit for calculating a first characteristic vector by performing signal processing on the muscle active period and calculating a second characteristic vector by performing signal processing on the motion duration;
A search unit for searching for a signal corresponding to the integrated characteristic vector in a database based on an integrated characteristic vector obtained by combining the first characteristic vector and the second characteristic vector; And
An output unit for outputting a text corresponding to the searched signal,
And a sign recognition system. The method according to claim 1,
Wherein the obtaining unit comprises:
A first receiving unit for receiving an EMG signal according to an operation of waving the user's wrist through a plurality of electrode channels included in the sensor measuring device;
A channel identification unit for identifying an electrode channel having a maximum effective output value among the plurality of electrode channels based on the electromyogram signal received from each of the plurality of electrode channels; And
An alignment unit for rearranging the plurality of electrode channels included in the sensor measurement device in consideration of the position of the identified electrode channel in the sensor measurement device,
≪ / RTI > The method according to claim 1,
Wherein the obtaining unit comprises:
A second receiving unit for receiving the inertia signal through an inertial measurement unit included in the sensor measuring device;
A calculation unit for calculating an orientation initial value of the inertia measurement unit using the inertia signal and an orientation value of the inertia measurement unit according to real-time reception of the inertia signal; And
A calculating unit calculating a roll angle, a pitch angle and a yaw angle of the inertia measuring unit using a reference vector determined based on the orientation initial value and a motion vector determined based on the orientation value,
≪ / RTI > The method according to claim 1,
The extracting unit extracts,
Extracting the muscle activity section by applying a teaser-kaiser energy operator (TKEO) technique to the EMG signals received from each of the plurality of electrode channels included in the sensor measurement device,
Wherein the motion interval is extracted by applying a Teeter-Kaiser Energy Operator (TKEO) technique to the inertia signal received from the inertial measurement unit included in the sensor measurement device. The method according to claim 1,
The extracting unit extracts,
Extracting, as the muscle activity period, a section that is equal to or greater than a preset muscle activity threshold value in the EMG signal,
And extracts, as the motion section, a section having a motion threshold value equal to or greater than a preset motion threshold value in the inertia signal. The method according to claim 1,
The calculating unit calculates,
Calculating the first characteristic vector by calculating an effective output value of the electromyogram signal for each of the plurality of electrode channels included in the sensor measurement device based on the muscle activity period,
And the second feature vector is calculated by applying a high pass filter based on the motion duration. The method according to claim 6,
Wherein the first feature vector and the second feature vector are resampled by normalizing the time data. The method according to claim 1,
The search unit may search,
Wherein the search is performed using a neural network formed through learning of a specific hydration operation. The method according to claim 1,
The sensor measuring device includes:
An arm band worn to surround the arms;
A plurality of electrodes disposed to be spaced apart from each other along the inner circumference of the armband so as to face the arms; And
An inertia measurement unit formed in one area of the sensor measurement device,
≪ / RTI >
Wherein the inertial measurement unit includes a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor. Obtaining an EMG signal and an inertia signal of the user from a sensor measuring instrument worn on a user's arm;
Extracting a muscle activity section from the EMG signal to detect a hydration operation taken by the user and extracting a motion section of the arm part from the inertia signal;
Calculating a first characteristic vector by performing signal processing on the muscle active section and calculating a second characteristic vector by performing signal processing on the motion section;
Searching for a signal corresponding to the integrated characteristic vector in a database based on an integrated characteristic vector obtained by integrating the first characteristic vector and the second characteristic vector; And
Outputting a text corresponding to the retrieved signal,
/ RTI > 11. The method of claim 10,
Wherein the acquiring comprises:
Receiving an EMG signal according to an operation of waving the wrist of the user through a plurality of electrode channels included in the sensor measurement device,
An electrode channel having a maximum effective output value among the plurality of electrode channels based on the EMG signal received from each of the plurality of electrode channels,
Wherein the plurality of electrode channels included in the sensor measurement device are rearranged in consideration of the position of the identified electrode channel in the sensor measurement device. 11. The method of claim 10,
Wherein the acquiring comprises:
Receiving the inertia signal via an inertial measurement unit included in the sensor measurement device,
Calculating an orientation initial value of the inertia measurement unit and an orientation value of the inertia measurement unit according to real-time reception of the inertia signal using the inertia signal,
A pitch angle and a yaw angle of the inertial measurement unit are calculated using a motion vector determined based on the reference vector determined based on the orientation initial value and the orientation value. 11. The method of claim 10,
Wherein the extracting comprises:
Extracting the muscle activity section by applying a teaser-kaiser energy operator (TKEO) technique to the EMG signals received from each of the plurality of electrode channels included in the sensor measurement device,
Wherein the motion interval is extracted by applying a Teeter-Kaiser Energy Operator (TKEO) technique to the inertia signal received from the inertial measurement unit included in the sensor measurement device. 11. The method of claim 10,
Wherein the extracting comprises:
Extracting, as the muscle activity period, a section that is equal to or greater than a preset muscle activity threshold value in the EMG signal,
And extracts, as the motion interval, a section having a motion threshold value equal to or greater than a preset motion threshold value in the inertia signal. 11. The method of claim 10,
Wherein the calculating step comprises:
Calculating the first characteristic vector by calculating an effective output value of the electromyogram signal for each of the plurality of electrode channels included in the sensor measurement device based on the muscle activity period,
Wherein the second feature vector is calculated by applying a high pass filter based on the motion duration. 16. The method of claim 15,
Wherein the first feature vector and the second feature vector are resampled by normalizing the time data. 11. The method of claim 10,
Wherein the searching comprises:
Wherein the search is performed using a neural network formed through learning of a specific hydration operation. 11. The method of claim 10,
The sensor measuring device includes:
An arm band worn to surround the arms;
A plurality of electrodes disposed to be spaced apart from each other along the inner circumference of the armband so as to face the arms; And
An inertia measurement unit formed in one area of the sensor measurement device,
≪ / RTI >
Wherein the inertia measurement unit includes a three-axis acceleration sensor, a three-axis angular velocity sensor, and a three-axis geomagnetic sensor. 19. A computer-readable recording medium having recorded thereon a program for executing the method of any one of claims 10 to 18 in a computer.

Images