WO2016144284A1

WO2016144284A1 - Method for recognising the movement and psycho-emotional state of a person and device for carrying out said method

Info

Publication number: WO2016144284A1
Application number: PCT/UA2016/000027
Authority: WO
Inventors: Елизавета Сергеевна ВОРОНКОВА
Original assignee: Елизавета Сергеевна ВОРОНКОВА
Priority date: 2015-03-06
Filing date: 2016-03-01
Publication date: 2016-09-15

Abstract

The group of inventions relates to electronic devices which are worn on the body and which are intended for control of and interaction with other electronic devices by means of recognition of the psycho-emotional state and movements of a person, and can be used as a means of communication between people; as a means of controlling electronic devices, prostheses, computer programmes, mobile applications for entertainment and work in a person's daily life, and also for monitoring a person's safety or the clinical condition of patients. For the purposes of the method for recognising the psycho-emotional state and movements of a person, a device is used in which the following motion sensors are accommodated: an accelerometer; a gyroscope; a magnetometer; and an electromyograph, which recognise hand and forearm gestures; and also psycho-emotional sensors for determining the state of a person: a psychogalvanic reflex; a rheogram; a photoplethysmogram. The data obtained from the sensors are processed using initial processing means accommodated within the device, and are transmitted, using wired or wireless technology, to devices for further processing and/or reproduction of the information.

Description

METHOD OF MOTION RECOGNITION AND PSYCHO-EMOTIONAL

HUMAN STATES AND DEVICE FOR ITS IMPLEMENTATION

The invention relates to electronic devices that are worn on the body and which are designed to control and interact with other electronic devices by recognizing the psycho-emotional state of a person and his movements, in particular gestures of the hand and forearm, and can be used as a means of communication between people, a means of controlling electronic devices, prostheses, computer programs, mobile applications for entertainment and human work in everyday life, as well as for security control human, clinical condition of patients.

To improve the interaction of a person with electronic devices, devices for recognizing human movements, including his movement in space, as well as gestures made both by his limbs and the body as a whole, are used. Motion recognition is the process of displaying physical movements in a virtual form for the purpose of their further processing and use.

A known method of gesture recognition is the use of additional information input / output devices, the connection of which with controlled devices is via wires, as well as electromagnetic, optical, sound and other radiation. The methods of human exposure to such devices can be mechanical impact (pressing and twisting buttons and levers, touching sensors, compression, etc.), moving on the surface (for example, a computer mouse and its additional elements - scroll wheels, trackballs), tilt , rotation around an axis, rotation in a certain direction. Existing systems have drawbacks in use, since they are tied to stationary objects and occupy human hands.

One way to recognize gestures is through optical perception of the position of the hands. To do this, the inventions US 6 128 003, US 8 768 006 and US 5 454 043 offer camera image processing, and in US 8 638 989 B2 Leap Motion proposes the use of various types of cameras, including infrared.

The disadvantages of such methods and devices is the need to bind the operator to the camera or point in space. In the invention US 6 128 003 it is proposed to use gloves, which is also not convenient when used. In everyday life, the usual inertial measurement unit is common, which, in particular, is used to recognize gestures in smartphones and other mobile devices. These include a kit with an accelerometer, gyroscope and magnetometer, which serves to improve the user interface. Some functions implemented by this set have become standard, for example, auto-rotate the screen image when the device is rotated accordingly, and some functions are used by software developers, including games. This kit is used in the method by placing it on the forearm of the arm and, as a result, obtaining data on the rotation and movement of the arm in space.

For the method of gesture recognition, a myograph is also used - a device that perceives acoustic, electromagnetic and vibration effects that occur in muscles during contraction. The method consists in the removal and analysis of the electromyogram of the forearm of the hand, which allows to perceive muscle contraction and, as a result, to recognize gestures of the hand and individual fingers. To obtain an electromyogram, non-invasive (external) electrodes are used, which can have a different design, size, composition and quantity. All sensors, electrodes, data processing and transmission equipment are compactly housed in a fabric housing worn on the forearm of the hand. The device is convenient in everyday use.

Patent WO 2004107962 A2 proposes the use of various sensors, including myographs, an accelerometer and a plethysmograph, for analyzing a person’s physiological state: respiration, its phases and breathing problems, measuring the amount of carbon dioxide in the air, cardiac indicators, including various phases of a cardiogram, pulse , ventricular volume, etc., blood flow, blood pressure, glucose concentration and others. Such medical cases when oxygen deficiency / oversaturation affect the posture and movement of the person, therefore, it is proposed to use accelerometers installed mainly on the torso and legs of a person to track them.

In the patent US 20140163412 to improve the performance of the myograph in terms of movement of a person or limb, which includes muscle, it is proposed to monitor the intensity of movements for appropriate noise filtering. In some experimental versions, an accelerometer, gyroscope and magnetometer are used for this. However, the invention does not use the specified set of sensors for detecting movements, only to stabilize the operation of the myograph during the movement of the device.

In the invention WO 2010034308 A2, a method for producing biometric data of a group of people during a certain period of time - a game, for which a motion recognition device is attached to each of its participants. The space of the participants' movements is determined by the game field, and information from the motion recognition devices is transmitted to the central server for processing. Sensors that can be used in devices are called 3-dimensional accelerometers, gyroscopes and magnetometers in a simple version, as well as pressure, temperature and physiological sensors that determine the carrier’s pulse, respiratory rate, skin moisture, inhaled air, the amount of oxygen in the blood, and / or myographs.

Closest to the claimed invention is the patent WO 2009138587 A2, 2010, in the description of which it is proposed to process information about the physical or mental activity of an individual to use for each person an individual device worn on the body and containing a certain number of sensors capable of obtaining data corresponding to physical or biological device media activity. The invention claims the presence of at least one motion sensor, at least one physiological sensor, and the use of at least one positioning system. Motion sensors include an accelerometer, a gyroscope and a magnetometer, among physiological sensors there may be sensors of muscle activity (electromyograph, EMG) and a blood flow sensor. The device is connected to radio communications, which allows you to manage the collected data.

However, this method is used to obtain and process the physiological data of a person or animal, in particular horses and athletes during physical and mental activity, without indicating the place of measurement, and its purpose does not declare recognition of movements or gestures of the carrier. Besides. Processing of data (except for playback) transmitted wirelessly occurs only on the remote device.

The basis of the invention is the task to improve the interaction of a person with electronic devices by creating a method for recognizing gestures and the psycho-emotional state of a person with integration in any operating system and program, the ability to control multiple systems simultaneously and use several devices with one processing device, in particular, recognize gestures of both hands.

The second task, which is the basis of the invention, is the creation of a universal device for recognizing gestures and the psycho-emotional state of a person for controlling and interacting with other electronic devices. The problem is solved by the fact that in the method of recognizing the movements and psychoemotional state of a person using sensors - an accelerometer, gyroscope, magnetometer and electromyograph, which are placed in the device that put on the forearm of the hand, according to the invention, the psychoemotional sensor is additionally placed in the device, and using accelerometer, gyroscope, magnetometer, electromyograph, perform recognition of gestures of the hand, fingers and forearm of a person’s hand, and psychoemotional sensor perform recognition of psi oemotsionalnogo human condition.

The data obtained from the sensors is processed using initial processing means located on the human body, including as part of a device on the forearm, and transmitted using wired or wireless technologies to a device for further processing and / or reproduction of information.

At the initial stage of data processing, it is necessary to amplify, filter, and digitize, at the next stages, the actual recognition of gestures and the psycho-emotional state of a person is performed, after which a response to a gesture, physiological or emotional state of a person, performed by an additional device for processing and reproducing information, can occur.

Depending on the complexity of the task and the recognition algorithm for various types of gestures or emotions, as well as the power of processing tools and the number of sensors used, different computing capabilities are necessary for the indicated stages of information processing, and, therefore, the number and power of processing devices can vary.

The results of information processing using means placed on a person’s hand can be displayed on the built-in interface elements or transmitted wirelessly to other elements for displaying the received information and / or controlling electronic devices.

The results of information processing using means located on different parts of the human body can be transmitted to the processing device using wires, the information processing results in this case are reflected on the built-in interface elements or transmitted via wireless connection to other display elements of the received information and / or control electronic devices.

As a device for further processing can use a mobile communication device - a smartphone.

As a device for further processing, they can use a personal computer or its analogues - a tablet, laptop, netbook. As a device for further processing, they can use a specially designed processing device with autonomous power supply, with the help of which they process sensor data and display the results of processed information about human movements on a monitor or other means of the interface - graphic, sound, vibration, temperature, mechanical, or use it to control electronic devices.

As a processing device, electronic production control systems, or vehicles, or musical instruments, or medical devices can be used.

As a psychoemotional sensor, a skin-galvanic reaction sensor can be used, with the help of which recognition of the psychoemotional state of a person is performed on the basis of data obtained on changes in physiological parameters - electrical conductivity of human skin or electrical potentials of the skin.

As a psycho-emotional sensor can use a rheogram sensor.

As a psycho-emotional sensor, a photoplethysmogram sensor can be used.

The psychoemotional state of a person can also be determined as a result of using data obtained from sensors - an accelerometer, gyroscope, magnetometer, electromyograph, given that a change in the psychoemotional state of a person is often reflected in his gestures or its absence.

Data on the psycho-emotional sensor can be used to clarify gesture recognition, for example, when the physiological indicators adopted by the psycho-emotional sensor indicate that the carrier is in the sleep phase, the gesture recognition function is disabled.

As a wireless technology for transmitting information using Bluetooth and / or Wi-Fi.

The second task is solved by the fact that a device for recognizing the movements and psycho-emotional state of a person, which is worn on the forearm of the hand and in the body of which sensors are placed - an accelerometer, a gyroscope, a magnetometer, an electromyograph, according to the invention, additionally contains a psycho-emotional sensor, initial processing means are placed on the human body sensor signals using wireless technology connected to a device for further processing and / or reproduction of information.

As a device for further processing can be used mobile communication device - smartphone.

As a device for further processing, a personal computer or its analogues - a tablet, laptop, netbook can be used.

As a further processing device, a specially designed self-powered processing device can be used.

As a device for further processing, electronic production control systems, or vehicles, or musical instruments, or medical devices can be used.

As a psycho-emotional sensor, a skin-galvanic reaction sensor can be used.

As a psycho-emotional sensor, a rheogram sensor can be used.

As a psycho-emotional sensor, a photoplethysmogram sensor can be used.

The device has autonomous battery power.

The inventive method and device improve the interaction of a person with a computer by using an accelerometer, gyroscope, magnetometer, electromyograph, perform recognition of gestures of fingers, hands and forearms of a person’s hand.

A distinctive feature of the invention is the use of an electromyograph specifically for gesture recognition.

The advantage of the device compared to the prototype is its versatility, namely, integration in any operating system and program, the ability to manage multiple systems at the same time, the ability to use several devices with one processing device, in particular, recognize gestures of both hands.

Thanks to its lightweight flexible design and fabric cuffs, the device is convenient for everyday use.

A psycho-emotional sensor (skin-galvanic reaction sensor, rheograph or photoplethysmograph) is used to recognize a person's psycho-emotional state. Based on the data on the galvanic (electrical) properties of the skin, blood flow in the vessels and its instant composition, we can draw approximate conclusions about the medical condition of the device’s carrier, including breathing or blood supply problems, as well as about the emotional state, for example, about strong emotions excitement / relaxation, fright, mood swings, etc. In addition to these functions, the psycho-emotional sensor can perform the function of a heart rate monitor, that is, measure the pulse of a carrier, and the corresponding information processing device can play a pulse of a carrier on a managed device or perform other appropriate actions when the pulse changes within predetermined limits.

When controlling electronic devices using gestures, the use of a psycho-emotional sensor improves the interface. For example, when managing a video game, a psycho-emotional sensor picks up the player’s strong negative emotions and, accordingly, can change the characteristics of the game.

The method allows during sports and other physical exercises to determine the intensity and nature of the athlete’s movements, the result of which can be data on the movements characteristic of a particular athlete, the carrier’s energy consumption, sweating and blood circulation intensities, as well as his psychoemotional state, as measured by a psychoemotional sensor. Thus, you can get information about the athlete’s physical capabilities, the body’s response to certain physical exertion, the psycho-emotional reaction to the elements of movement, the athlete’s mood and health risk at a particular time, fatigue, activity, and the like. The indicators of the psychoemotional sensor allow you to catch the body's reaction to the use by the body of medical and other substances, in particular doping, narcotic and alcoholic substances. In addition, the measurements of the electromyograph can be used to control the technique for performing specific movements of the hand and forearm, restoring the limb after injuries and other physiological indicators. Thus, the invention can also serve as the basis for the manufacture and improvement of prosthetic arms and control their use.

The inventive method also allows you to obtain data on the movements of a group of people, participants in sports and other games, and the use of a psycho-emotional sensor significantly increases the possibility of its appointment. In addition to recognizing the movements of several persons and the possibility of processing them, one can observe the psycho-emotional state of the player, his reaction to events, as well as other players, increases the interest in observing the game, controlling the players by their coach, simplifies modeling of game situations and monitoring violations, etc.

The invention is illustrated by drawings: where in Fig. 1 is a diagram of a device for recognizing movements and psycho-emotional state of a person, Fig. 2 is a variant of the appearance of a device for recognizing movements and psycho-emotional human condition.

The device is worn on the forearm of the hand. Sensors are located in the device’s body - accelerometer 1, gyroscope 2. magnetometer 3, electromyograph 4, psychoemotional sensor 5 (skin-galvanic reaction sensor, rheograph or photoplethysmograph), as well as device 6 for initial processing of sensor signals and wireless data transmission modules. As a device for further processing and / or reproduction of information transmitted using wireless technologies, a mobile communication device - a smartphone can be used; personal computer or its analogues - tablet, laptop, netbook; specially designed self-powered processing device; electronic production control systems, or vehicles, or musical instruments, or medical devices.

The device has autonomous battery power (not shown).

A method of recognizing the movements and psycho-emotional state of a person is as follows.

A device for recognizing the movements and psycho-emotional state of a person is put on the forearm of the hand. Using sensors - accelerometer 1, gyroscope 2, magnetometer 3, electromyograph 4 perform recognition of gestures of fingers, hands and forearms of a person’s hand. Physical movement data obtained from sensors 1, 2, 3, 4 is processed using processing tools, which are located:

- directly on the upper limb (arm) of a person, while the results of information processing in this case are reflected on the built-in interface elements or transmitted via wireless connection to other elements for displaying and using the received information;

- on other parts of the human body, while the data from the sensors to the processing device 6 is transmitted using wires, and the information processing results in this case are reflected on the built-in interface elements or transmitted via wireless connection to other elements for displaying and using the obtained information.

The obtained data is transmitted using wired or wireless technologies (Bluetooth and Wi-Fi) to the device 7 for further processing (mobile communication device - smartphone; personal computer or its analogues - tablet, laptop, netbook; specially designed processing device with autonomous nutrition; electronic production control systems, or vehicles, or musical instruments, or medical devices). In this case, a specially designed processing device with autonomous power supplies processes the sensor data and displays the results of the processed information about human movements on a monitor or other means of the interface - graphic, sound, vibration, temperature, mechanical and / or uses them to control electronic devices.

When using the psychoemotional 5 sensor (skin-galvanic reaction sensor, rheograph or photoplethysmograph), the psychoemotional state of a person is recognized based on the data obtained on a change in one of the following physiological indicators - the electrical conductivity of human skin; electrical potentials of the skin; blood vessels of the skin.

Under the psychoemotional state of a person recognized with the help of the psychoemotional 5th sensor, we mean a temporary or prolonged state of psychoemotional arousal or inhibition caused by exposure to a person from outside, a change in his state of health, the use of alcoholic, narcotic drugs or other psychostimulating drugs, and affects his physiological indicators.

The psycho-emotional state can also be determined as a result of using data obtained from sensors 1, 2, 3, 4, because a change in the psycho-emotional state of a person is often reflected in its gestures or its absence. Similarly, data from the psycho-emotional 5 sensor can be used to clarify gesture recognition, for example, physiological indicators received by the sensor indicate that the carrier is in the sleep phase, so the gesture recognition function can be turned off.

To remove the electromyogram and skin-galvanic reaction from the forearm, non-invasive (external) electrodes are used that directly come into contact with the skin. A feature of the device is the use of common electrodes for the electromyograph 4 and psychoemotional 5 sensor, the number of which may differ depending on the use of specific muscles and the number of desired movements.

Claims

CLAIM

1. A method for recognizing the movements and psycho-emotional state of a person using sensors - an accelerometer, gyroscope, magnetometer and electromyograph placed in a device that is worn on the forearm of the hand, characterized in that the device additionally contains a psycho-emotional sensor using an accelerometer, gyroscope, magnetometer, electromyograph recognition of gestures of the hand and forearm of a person’s hand, and psychoemotional sensor perform recognition of the psychoemotional state of a person, after that the data obtained from the sensors is processed using the initial processing means placed on the forearm as part of the device, and transmitted using leading or wireless technologies to further processing devices and / or information reproduction.

2. The method according to claim 1, characterized in that the results of information processing using means placed on a person’s hand are displayed on the built-in interface elements or transmitted via wireless connection to other processing and / or display elements of the received information.

3. The method according to claim 1, characterized in that the information processing results using means located on different parts of the human body are transmitted to the processing device using wires, the information processing results in this case are displayed on the built-in interface elements or transmitted wirelessly connections to other elements of display and use of the information received.

4. The method according to claim 1, characterized in that as the processing device use a mobile communication device - a smartphone.

5. The method according to claim 1, characterized in that a personal computer or its analogs — a tablet, laptop, netbook — is used as a processing device.

6. The method according to claim 1, characterized in that a specially designed device with autonomous power supply is used as a processing device, with the help of which the sensor data are processed and the results of the processed information about human movements are displayed on a monitor or other means of the interface - graphic, sound, vibration , temperature, mechanical, or use it to control electronic devices.

7. The method according to claim 1, characterized in that the processing device uses electronic production control systems, or vehicles, or musical instruments, or medical devices.

8. The method according to claim 1, characterized in that the sensor of the skin-galvanic reaction is used as a psychoemotional sensor, with the help of which recognition of the psychoemotional state of a person is performed on the basis of the obtained data on changes in physiological parameters - electrical conductivity of human skin or electrical potentials of the skin.

9. The method according to claim 1, characterized in that a rheogram sensor is used as a psycho-emotional sensor.

10. The method according to claim 1, characterized in that the photoplethysmogram sensor is used as a psychoemotional sensor.

1 1. The method according to claim 1, characterized in that, together with the recognition of the psychoemotional state of a person, the psychoemotional sensor also serves as a heart rate monitor.

12. The method according to claim 1, characterized in that the psycho-emotional state of a person is also determined as a result of using data obtained from sensors - an accelerometer, gyroscope, magnetometer, electromyograph, given that a change in the psycho-emotional state of a person often affects his gesticulation or its absence.

13. The method according to claim 1, characterized in that the data from the psycho-emotional sensor is used to clarify the recognition of gestures, in particular, when the physiological indicators adopted by the psycho-emotional sensor indicate that the carrier is in the sleep phase, the gesture recognition function is disabled.

14. The method according to claim 1, characterized in that as wireless technologies use Bluetooth and Wi-Fi.

15. A device for recognizing the movements and psycho-emotional state of a person, which is worn on the forearm of the hand and in the body of which there are sensors - an accelerometer, gyroscope, magnetometer, electromyograph, characterized in that the device additionally contains a psycho-emotional sensor, all sensors are connected to the initial signal processing device as part devices on the forearm and using wired or wireless technologies are connected to the device for further processing and / or reproduction of information.

16. The device according to 14, characterized in that as the processing device used a mobile communication device - a smartphone.

17. The device according to 14, characterized in that a personal computer or its analogues - a tablet, laptop, netbook, is used as a processing device.

18. The device according to p. 14, characterized in that as a processing device used specially designed processing device with autonomous power.

19. The device according to 14, characterized in that as the processing device used electronic production control systems, or vehicles, or musical instruments, or medical devices.

20. The device according to 14, characterized in that as a psycho-emotional sensor used sensor skin-galvanic reaction.

21. The device according to 14, characterized in that the rheogram sensor is used as a psycho-emotional sensor.

22. The device according to 14, characterized in that as a psycho-emotional sensor used sensor photoplethysmogram.

23. The device according to 14, characterized in that it has autonomous power from the battery.