KR20200075107A - Wearable device and method for recognizing gesture thereof - Google Patents

Abstract

According to the present invention, a wearable device capable of recognizing gestures comprises: a flexible epithelial tactile based array sensor configured to be detachable and attachable while measuring change data of electrical resistance according to movements of a wrist; a memory in which a program for recognizing gestures is stored; and a processor executing a program stored in the memory, wherein the processor, upon execution of the program, measures movements of a first body part of the wrist through a first sensor of the array sensor, measures movements of a second body part of the wrist through a second sensor of the array sensor, and recognizes gestures of a user based on the change data of the electrical resistance measured from the array sensor.

Description

Wearable device and its gesture recognition method {WEARABLE DEVICE AND METHOD FOR RECOGNIZING GESTURE THEREOF}

The present invention relates to a wearable device and a gesture recognition method thereof, and more specifically, a wearable device capable of acquiring a biosignal from a clip sensor based on a flexible epithelial tactile sensor array, and learning and recognizing a gesture based thereon. Recognition method.

Recently, as technology development and commercialization of wearable devices in addition to smart phones have become active, interest in gesture recognition technology is also increasing.

Gesture recognition technology can be roughly divided into a contact method and a non-contact method.

The contact method is a method of attaching a sensor or a device to a user's body part and utilizing the data obtained therefrom for motion recognition.

In addition, the non-contact method is a method of acquiring a user's movement as an image using a camera and recognizing a gesture by utilizing it.

Meanwhile, a gesture recognition technology using a wearable device mainly uses a contact method, and acquires data by attaching a sensor or device capable of detecting the movement of the user to be worn on a person's body or clothes.

In this case, there is an inconvenience in that the user has to wear a wearable device directly, but since the sensor is directly attached, relatively accurate operation information can be obtained.

Sensors used in wearable devices for gesture recognition mainly include IME (Inertial Measurement Unit) sensors, EMG sensors, gyro sensors, and the like.

However, since these sensors are difficult to miniaturize, they are not suitable for wearable devices such as smart watches or smart bands, and in particular, there is a problem that they are difficult to wear on curved body parts such as wrists or forearms.

According to an embodiment of the present invention, a wearable device capable of learning and recognizing a gesture by acquiring a biosignal according to a movement change of a wrist muscle using an array sensor that can be coupled with a wearable device, and analyzing the biosignal to extract feature points of the gesture It is intended to provide a device and a gesture recognition method thereof.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the wearable device capable of gesture recognition according to the first aspect of the present invention measures the change data of electrical resistance according to the movement of the wrist and is a flexible epithelium tactile configured to be detachable and attachable. It includes a base array sensor, a memory in which a program for gesture recognition is stored, and a processor for executing a program stored in the memory. At this time, as the processor executes the program, the movement of the first body part of the wrist is measured through the first sensor of the array sensor, and the movement of the second body part of the wrist is measured by the array sensor. Measured by the second sensor, and recognizes the user's gesture based on the change data of the electrical resistance measured from the array sensor.

The processor may calculate an absolute difference value between two consecutive values of the change data of the electrical resistance, and recognize a gesture according to the movement of the user's wrist based on the calculated absolute difference value.

The processor may calculate a peak value for each movement of the muscle according to the movement of the user's wrist from the change data of the electrical resistance, and distinguish a user's gesture based on the peak value for each movement of the muscle.

The processor learns a recognition algorithm by using the SVM (Support Vector Machine) for the calculated peak value for each movement of the muscle, inputs the change data of the electrical resistance to the learned recognition algorithm, and outputs the result, and outputs the output. If the probability of the corresponding gesture is calculated according to the result obtained and indicates a predetermined probability or more, it can be recognized as a gesture.

The array sensor may measure the change data of the electrical resistance as the deformation of the degree of distortion during contraction or relaxation according to the movement of the wrist occurs.

In addition, the gesture recognition method using the array sensor that can be combined with the wearable device according to the second aspect of the present invention includes measuring the change data of the electrical resistance according to the movement of the wrist using the array sensor and the change data of the electrical resistance And recognizing the user's gesture based on the first step.

The array sensor may be a flexible epithelial tactile sensor array.

The step of measuring the change data of the electrical resistance according to the movement of the wrist using the array sensor acquires the change data of the electrical resistance as the deformation of the degree of distortion during contraction or relaxation according to the movement of the wrist occurs. can do.

The measuring of electrical resistance change data according to the movement of the wrist using the array sensor may include measuring a movement of the first body part of the wrist through a first sensor and of the second body part of the wrist. It may include the step of measuring the movement of the second sensor.

Recognizing a user's gesture based on the change data of the electrical resistance includes: calculating an absolute difference between two consecutive values of the change data of the electrical resistance; And recognizing a gesture according to the movement of the user's wrist based on the calculated absolute difference value.

Recognizing a gesture according to the movement of the user's wrist based on the calculated absolute difference value, calculates a peak value for each movement of the muscle according to the movement of the user's wrist from the change data of the electrical resistance, the The user's gesture can be distinguished based on the peak value for each movement of the muscle.

One embodiment of the present invention, further comprising the step of learning a recognition algorithm using the support vector machine (SVM) for the calculated peak value for each movement of the muscle, the user's gesture based on the change data of the electrical resistance In the step of recognizing, a change result of the electrical resistance is input to the learned recognition algorithm, a result is output, and a probability of a corresponding gesture is calculated according to the output result to recognize a gesture as a gesture. Can.

According to any one of the above-described problem solving means of the present invention, a sensor having a flexible property applicable to a wearable device may be used to acquire a biosignal according to a change in wrist muscle movement to learn and recognize a gesture.

In particular, it is easy to wear and can be applied to various wearable devices by using a naturally bendable property, and it is possible to generate and use individual gestures for each user by learning and recognizing various motion motions.

1 is a block diagram of a wearable device according to an embodiment of the present invention.
2 is a view for explaining an array sensor applied to an embodiment of the present invention.
3 is a diagram illustrating an example of an array sensor.
4 is a diagram illustrating an example in which an array sensor is worn and attached.
5 is a flowchart of a gesture recognition method according to an embodiment of the present invention.
6 is an example of an operation that can be made using a wrist and a finger.
7 is electrical resistance change data measured according to the operation of FIG. 6.
8 is a diagram illustrating six gestures defined in an embodiment of the present invention.
9 is a matrix showing experimental results based on the gesture defined in FIG. 8.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, parts not related to the description are omitted in the drawings to clearly describe the present invention.

When a part of the specification "includes" a certain component, it means that other components may be further included instead of excluding other components unless specifically stated otherwise.

1 is a block diagram of a wearable device 100 according to an embodiment of the present invention. 2 is a view for explaining an array sensor 110 applied to an embodiment of the present invention. 3 is a diagram illustrating an example of an array sensor 110. 4 is a diagram illustrating an example in which the array sensor 110 is worn and attached.

The wearable device 100 capable of gesture recognition according to an embodiment of the present invention includes an array sensor 110, a memory 120, and a processor 130.

The array sensor 110 measures data of a change in electrical resistance according to the movement of the wrist, and is configured to be detachable and attachable.

In this case, the array sensor 110 in one embodiment of the present invention may be a flexible epidermal tactile sensor array (FETSA).

The array sensor 110 measures the physical deformation of the sensor according to the movement of the wrist. That is, the sensor measures a change in electrical resistance according to muscle movement and acquires it as data.

Referring to Figure 2, the various muscles on the wrist contract and relax while the wrist is moving. At this time, the thickness or angle of the wrist, the degree of distortion, etc. changes, and accordingly, the degree of distortion of the sensor wound around the wrist changes. As a result, deformation of the strain gage of the sensor occurs and a change in resistance is acquired.

Accordingly, the array sensor 110 may measure the change data of the electrical resistance as the deformation of the warp during contraction or relaxation due to movement of the wrist occurs.

In one embodiment of the present invention, the flexible epithelial tactile based array sensor 110 may be implemented as shown in FIG. 3.

In the case of FIG. 3, the clip type array sensor 110 for attaching to a smart watch is composed of a flexible substrate that is joined to a bracelet part and wound around a user's wrist, and a metal strain gauge (Metal) on the upper surface thereof. strain gauge) can be attached.

In addition, a portion facing the flexible substrate includes a unit for acquiring data on resistance change according to the strain gauge.

On the other hand, the array sensor 110 may include a plurality of sensors, each sensor is located in the relevant muscle to observe the muscle activity responsible for the movement of the wrist. That is, the plurality of sensors may include a first sensor that measures motion in the first body part and a second sensor that measures motion in the second body part.

For example, as illustrated in (a) of FIG. 4, the first sensor may be located in an abductor pollicis longus muscle responsible for the movement of the thumb.

In this case, the first body part and the second body part mean different body parts, and the number of sensors included in the array sensor 110 may be provided in various numbers as necessary.

A program for gesture recognition is stored in the memory 120, and the processor 130 executes the program stored in the memory 120. Here, the memory 120 is a non-volatile storage device and a volatile storage device that keeps stored information even when power is not supplied.

For example, the memory 120 is a compact flash (CF) card, a secure digital (SD) card, a memory stick, a solid-state drive (SSD), and a micro SD NAND flash memory such as cards, magnetic computer storage devices such as hard disk drives (HDDs), and optical disc drives such as CD-ROMs, DVD-ROMs, and the like. Can.

For reference, the components shown in FIG. 1 according to an embodiment of the present invention may be implemented in software or in a hardware form such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and perform predetermined roles. can do.

However,'components' are not limited to software or hardware, and each component may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors.

Thus, as an example, a component is a component, such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, sub Includes routines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays and variables.

Components and functions provided within those components may be combined into a smaller number of components or further separated into additional components.

Hereinafter, a gesture recognition method using an array sensor 110 that can be combined with a wearable device 100 according to an embodiment of the present invention will be described with reference to FIGS. 5 to 7.

5 is a flowchart of a gesture recognition method according to an embodiment of the present invention. 6 is an example of an operation that can be made using a wrist and a finger. 7 is electrical resistance change data measured according to the operation of FIG. 6.

First, the change data of the electrical resistance according to the movement of the wrist is measured using the array sensor 110 (S110).

At this time, the array sensor 110 is composed of a plurality of sensors, it is possible to acquire the data of the change in electrical resistance as the deformation of the degree of distortion during contraction or relaxation according to the movement of the wrist occurs.

Next, the user's gesture is recognized based on the change data of the electrical resistance (S120).

Specifically, in order to confirm whether the signal acquired from the array sensor 110 changes according to the movement of the wrist, the processor calculates a peak value for each movement of the muscle according to the movement of the user's wrist from the change data of the resistance, The user's gesture may be distinguished based on the calculated peak value for each movement of the muscle.

Referring to FIG. 6, there are 8 large movements that a human can make using a wrist and a finger.

Since these motions theoretically use different muscles, different signals must be observed for each motion in the array sensor 110.

FIG. 7 relates to data of a change in electrical resistance in the array sensor 110 according to eight wrist movements in FIG. 6, and shows a change during a gesture in the case of a shaded area.

According to FIG. 7, it can be confirmed that when the user performed the eight operations of FIG. 6, the changes of each of the four sensors are different.

As described above, since the resistance value of the array sensor 110 worn on the wrist changes according to the movement of the wrist, and different signals are obtained according to the movement as shown in FIG. 7, a characteristic of reflecting the change of the signal to distinguish the gesture Use

At this time, an embodiment of the present invention calculates the absolute difference value of two successive values of DAMV (Difference Absolute Mean Value), that is, data of change in electrical resistance, according to Equation 1 below, and uses the calculated absolute difference as a feature Gestures according to the user's wrist movement.

[Equation 1]

Meanwhile, in the gesture recognition method according to an embodiment of the present invention, a process of learning a recognition algorithm may be additionally performed by using the SVM (Support Vector Machine) to calculate the calculated peak value for each movement of the muscle.

That is, in the learning process, in order to learn the signal when the user's movement occurs, the peak value of the signal is extracted and the recognition algorithm is learned using the peak value of the signal.

And after the learning is completed, the data of the change in electrical resistance corresponding to the characteristic in the signal received from the array sensor 110 is extracted, and inputted to the learned recognition algorithm to output the result.

Then, when the probability of the corresponding gesture is calculated according to the output result and indicates a predetermined probability or more, the gesture may be finally recognized.

At this time, the probability of the gesture may be calculated according to Equation (2).

[Equation 2]

For example, when the preset probability is 80%, when the probability of the recognized gesture exceeds 80%, the corresponding case may be recognized as a gesture.

In this case, when there are a plurality of cases exceeding a preset probability, a case having the highest probability can be recognized as a gesture.

If all cases do not exceed a preset probability, the gesture may be regarded as not detected.

In the above description, steps S110 to S120 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted if necessary, and the order between the steps may be changed. In addition, even if omitted, the contents already described with respect to the wearable device 100 in FIGS. 1 to 4 are also applied to the gesture recognition methods of FIGS. 5 to 7.

In order to verify the performance of the present invention, a gesture recognition experiment was conducted, which will be described with reference to FIGS. 8 and 9.

8 is a diagram illustrating six gestures defined in an embodiment of the present invention. 9 is a matrix showing experimental results based on the gesture defined in FIG. 8.

For the experiment, six gestures were defined as shown in FIG. 8. In Figure 8 (1) is a finger pinch (pinch), (2) and (3) is the relaxation and contraction of the fingers, (4) and (5) are the wrist bends, and finally (6) is the warp of the wrist. It was defined as

As a result of experimenting with these six gestures, as shown in FIG. 9, it was confirmed that the average recognition rate was 97.8% among the total of 1440 recognition experiments.

One embodiment of the present invention may also be implemented in the form of a computer program stored on a medium executed by a computer or a recording medium including instructions executable by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism, and includes any information delivery media.

Although the methods and systems of the present invention have been described in connection with specific embodiments, some or all of their components or operations may be implemented using a computer system having a general purpose hardware architecture.

The above description of the present invention is for illustration only, and a person having ordinary knowledge in the technical field to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and it should be interpreted that all changes or modified forms derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present invention. do.

100: wearable device
110: array sensor
120: memory
130: processor

Claims (12)

In a wearable device capable of gesture recognition,
Flexible epithelial tactile-based array sensor configured to measure and change the electrical resistance according to the movement of the wrist and to be attached and detached
Memory for storing the program for the gesture recognition and
It includes a processor for executing a program stored in the memory,
As the processor executes the program, the movement of the first body part of the wrist is measured through the first sensor of the array sensor, and the movement of the second body part of the wrist is measured by the array sensor. A wearable device that measures through a sensor and recognizes a user's gesture based on the change data of the electrical resistance measured from the array sensor. According to claim 1,
The processor calculates an absolute difference value between two consecutive values of the change data of the electrical resistance, and recognizes a gesture according to the movement of the user's wrist based on the calculated absolute difference value. According to claim 2,
The processor calculates a peak value for each movement of the muscle according to the movement of the user's wrist from the change data of the electrical resistance, and distinguishes a user's gesture based on the peak value for each movement of the muscle. The method of claim 3,
The processor trains a recognition algorithm to use the SVM (Support Vector Machine) to calculate the calculated peak value for each movement of the muscle,
A wearable device that recognizes the gesture as a gesture when the change data of the electrical resistance is input to the learned recognition algorithm to output a result and calculates a probability of a corresponding gesture according to the output result to indicate a predetermined probability or more. According to claim 1,
The wearable device of the array sensor is to measure the change data of the electrical resistance as the deformation of the degree of warp during contraction or relaxation according to the movement of the wrist occurs. In the gesture recognition method using an array sensor that can be combined with a wearable device,
Measuring data of a change in electrical resistance according to the movement of the wrist using the array sensor, and
And recognizing a user's gesture based on the change data of the electrical resistance. The method of claim 6,
The array sensor is a flexible epithelial tactile sensor array gesture recognition method. The method of claim 6,
The step of measuring the change data of the electrical resistance according to the movement of the wrist using the array sensor,
Gesture recognition method for acquiring the change data of the electrical resistance as the deformation of the degree of distortion during contraction or relaxation according to the movement of the wrist occurs. The method of claim 6,
The step of measuring the change data of the electrical resistance according to the movement of the wrist using the array sensor,
Measuring movement of the wrist on a first body part through a first sensor; and
And measuring movement of the wrist on a second body part through a second sensor. The method of claim 6,
Recognizing a user's gesture based on the change data of the electrical resistance,
Calculating an absolute difference value between two successive values of the electrical resistance change data;
And recognizing a gesture according to the movement of the user's wrist based on the calculated absolute difference value. The method of claim 10,
Recognizing a gesture according to the movement of the user's wrist based on the calculated absolute difference value,
The gesture recognition method is to calculate a peak value for each movement of the muscle according to the movement of the user's wrist from the change data of the electrical resistance, and to distinguish a user's gesture based on the peak value for each movement of the muscle. The method of claim 11,
Further comprising the step of learning the recognition algorithm using the SVM (Support Vector Machine) for the calculated peak value for each movement of the muscle,
Recognizing a user's gesture based on the change data of the electrical resistance,
The gesture recognition method is to input a change data of the electrical resistance to the learned recognition algorithm to output a result, and to calculate a probability of a corresponding gesture according to the output result to indicate a predetermined probability or more.

Images