WO2019083405A1

WO2019083405A1 - Virtual reality glove

Info

Publication number: WO2019083405A1
Application number: PCT/RU2018/000630
Authority: WO
Inventors: Андрей Сергеевич КАМОЦКИЙ
Original assignee: Федоров, Александр Владимирович
Priority date: 2017-10-27
Filing date: 2018-09-28
Publication date: 2019-05-02

Abstract

The invention relates to manipulators, namely to virtual gloves intended for working with interactive devices. The claimed device comprises sensors arranged in the fingers of the glove, the sensors being connected to a motherboard. The sensors used are IMU sensors which are arranged within a glove and made of fabric, each of the IMU sensors containing an accelerometer, gyroscope and magnetometer. Four IMU sensors are arranged on the next-to-last phalanges of the little finger, ring finger, middle finger and index finger; two IMU sensors are arranged on the first and second phalanges of the thumb, one IMU sensor is arranged on the motherboard. Vibration motors are arranged on the glove fingers, one on each finger, wherein on all fingers apart from the thumb, vibration motors are secured on the next-to-last phalanges of the fingers while on the thumb a vibration motor is secured on the last phalanx. The motherboard, where the computer is arranged, is arranged on the glove. IMU sensors, vibration motors and s battery are connected to the motherboard by wires, and the integrated module comprising an IMU sensor and a light detector is arranged on the wrist of the glove.

Description

VIRTUAL REALITY GLOVE

DESCRIPTION

The device relates to manipulators, namely to virtual gloves,

designed to work with interactive devices, computer robotics and computer.

The prior art discloses a solution KR100221335, which describes a system and method for a glove to transmit a sensor value in a virtual reality system, including a series of sensors measuring movements of the wrist and fingers and outputting an electrical signal; The multiplexer selects the number of sensor signals included in one of the sensor groups. The purpose of the patent is fundamentally different from the stated solution. The analogs describe exactly the high-speed data transfer between the sensors and the virtual reality system. In the claimed solution, the data transfer method is irrelevant.

CN 1480822 describes a data entry device for people with

limited capacity, representing a glove having

a variety of sensors that can detect the movement of fingers, much faster than is usually required for pressing keys in a conventional keyboard; The sensor contains a virtual key detector and a key decoder, both of which are individually calibrated. The purpose of the glove and the detection method differ. In the analogue, it is proposed to interact by touching

For certain sensors, gloves for people with disabilities, touches will be transmitted to an external device. Our stated decision determines precisely the movements and positions of fingers and hands for transmission to an external device.

In the application US20150002401 described alternative gloves based on the "key" for mobile devices, including at least one pair of gloves, a plurality of sensors located on the first side, at least two screens located on each side of the glove. The task of the glove is to enter some keys for mobile devices. In this case, the glove does not detect the movement and position of the fingers and hand, as does the claimed solution.

In the patent US6870526 described glove mouse with virtual cursor tracking, designed to control the movement of the cursor and ensure the function of a computer mouse. The presented glove is intended only for controlling the cursor and the mouse.

Our solution can not only control the mouse cursor, but also

determine the movement of all fingers and hands to create a digital 3D model of the hand in the virtual space.

In the patent US7205979 described the device generating control signals for manipulating virtual objects in a computer system in

according to operator gestures or other parts of the body; The device includes a glove that includes sensors for detecting hand gestures, as well as hand position sensors connected to the glove and connected to a computer system for

determine the position of the hand in relation to the system.

Differs the principle of determining hand gestures. In analogue, gestures are determined by means of external sensors installed, for example, on a computer monitor.

The solution we declared uses inertial sensors built right into the glove.

The solution EP2624238 describes a virtual mock-up with a tactile hand, having simulated objects that are manipulated with the aid of hand wearable gloves during the operation of the simulator. The fingers on the fingers are equipped with 3D general motion detection tools integrated with at least one sensor for interaction tasks and one tactile component.

The analog is not specified as it is precisely the motion sensors used.

In the claimed solution, the use of inertial sensors with an accelerometer and a gyroscope is specified.

In international application WO98050839, a gesture-based data management system is described for processing real-time computer animation, such as virtual reality, and is controlled on a computer. The system includes a digital glove for data management based on operator hand gestures. The decision is based on the use of materials that change their resistance when bent.

The difference between the declared utility model is the use of inertial sensors.

In the international application WO2007129663 (publ .: 15.11.2007), an input device using sensors mounted on a finger is described. The user wears gloves with fingers, each of which has a motion detection sensor and a tactile sensor for each of the finger tips on both hands. Movement information at each of the fingertips is transmitted to various

information devices, such as a personal computer (PC), a pocket computer, and a cell phone, so that the positions of the corresponding fingertips are displayed on a virtual keyboard on the display of one of the information-processing devices.

The analog is not specified as it is precisely the motion sensors used.

The closest analogue is a virtual glove according to patent US9060385, publ .: 06/16/2015. The prototype describes a virtual reality glove containing sensors, gloves located on the fingers, and hand position sensors,

located on the phalanges of the fingers gloves, and the sensors are connected to the microprocessor.

The disadvantage of the prototype and other known solutions is the following.

As you know, a tactile analyzer has a high ability to

spatial localization. Its characteristic feature is the rapid development of adaptation (habituation), i.e. loss of feeling of touch or pressure.

The adaptation time depends on the strength of the stimulus, for different parts of the body it varies from 2 to 20 seconds. Thanks to the adaptation, we do not feel the touch of clothes to the body. See [Ekzertseva Ekaterina Vadimovna, Topic 1.1. General issues of life safety, Russian State Technological University. K.E. Tsiolkovsky (MATI), Lectures,

http://www.studfiles.ru/preview/854779/page:6/] Similar problems with working with virtual gloves are also caused by well-known counterparts, which reduces the effect of perception on the user over a certain period of work in them.

In addition, the technical problem of the known solutions when using IMU sensors without additional external tracking is the problem of drift - the accumulation of gyro / inertial sensor errors, a significant deviation of the obtained data from the real ones and, as a result, the impossibility of precise absolute positioning.

Problem solving is the elimination of these problems.

The technical result is the ability to receive and transmit data on the position of fingers, hands, elbow and shoulder joints to the computer or other device, as well as to carry out tactile feedback by transmitting vibrations to the fingers.

Also, the technical result is the ability to fix a virtual model of the hand not in the elbow joint, but in the shoulder and with high accuracy to recognize and transfer to the computer or other device all possible hand movements, including in the horizontal plane.

Another advantage over other known devices based only on external tracking is the ability to use outside stationary equipped rooms, which is convenient when using the solution with mobile devices (smartphones) or portable solutions of virtual and augmented reality.

This technical result is achieved due to the fact that the virtual reality glove is declared, which contains sensors, gloves located on the fingers, and the sensors are connected to the system board, characterized in that IMU sensors located inside the glove made of fabric are used as sensors, each IMU sensors contain an accelerometer, a gyroscope and a magnetometer, with four IMU sensors located on the penultimate phalanges little finger, ring, middle and index fingers, two IMU sensors located on the first and second phalanges of the thumb, one IMU sensor located on the system board; on the fingers of the gloves are mounted vibration motors, one on each finger, and on all fingers except the big one, the vibration motors are fixed on the penultimate phalanges of the fingers, and on the thumb a vibration motor is fixed on the last phalanx; the glove holds the motherboard where the computing module is located; IMU sensors, vibration motors, battery connected to the motherboard through wires; Combined module with IMU sensor and light sensor located on the wrist of the glove.

If necessary, the motherboard and battery can be made in a single package of translucent plastic.

Photodiodes are installed on the motherboard, which are connected to a combined module containing a photodiode and an IMU sensor.

Preferably, the photodiodes are mounted on the system board at its edges.

equidistant.

Preferably, four photodiodes are mounted on the system board.

An additional shoulder module can be installed on the shoulder, and connected to the system board using a wire through a combined module, which is made containing a connecting connector.

An IMU sensor can be installed inside the shoulder module.

The shoulder module can be made containing a wireless communication module and configured to connect to the system board over the air, and a battery is also located in the shoulder module.

The shoulder module, alternatively, contains a microcontroller and strain gauges connected to it.

Brief Description of the Drawings

Figure 1 shows a diagram of the device gloves using on the wrist and IMU sensor and without the use of photodiodes.

Figure 2 shows a diagram of the device gloves using photodiodes and located on the wrist of the combined module containing the photodiode and the IMU sensor

Fig. 3 shows a diagram of the device of the glove without using photodiodes in the system board and located on the wrist of the combined module containing the photodiode and IMU sensor.

Figure 4 shows the connection diagram to the glove of the shoulder module.

In the drawings: 1 - glove, 2 - inertial sensor (sensor) IMU, combining accelerometer, gyroscope and magnetometer, 3 - vibrating motor, designed to transmit vibration sensations, 4 - motherboard, 5 - battery, 6 - wires, 7 - the case that combines the motherboard and battery, 8 - photodiode, 9 - combined module, 10 - shoulder module, 11 - connecting wire between the combined module and the shoulder module.

The device (Figure 2) consists of a glove (1) made of a fabric inside which there are: IMU sensors (2), each of which contains an accelerometer, a gyroscope and a magnetometer, with four IMU sensors (2) located on

the penultimate phalanxes of the little finger, ring, middle and index fingers, two IMU sensors (2) are located on the first and second phalanges of the thumb, one IMU sensor (2) is located on the system board.

The fingers of the gloves (1) also have vibration motors (3) fixed, one on each finger, and on all fingers except the big one, the vibration motors (3) are fixed on the penultimate phalanges of the fingers, and on the thumb a vibration motor (3) is fixed on the last phalanx.

The motherboard (4) is fixed on the glove (1), where the computing module is located.

At the same time, the Bluetooth wireless communication module, for communication with a computer or other device via radio, is installed on the motherboard (4).

The motherboard (4) is mounted on top of the battery (5), which is mounted on the glove (1).

IMU sensors (2), vibration motors (3), battery (5) are connected to the system board (4) via wires (b).

If necessary, the motherboard (4) and battery (5) can be made in single body (7) of translucent plastic.

On the system board (4) there are also installed photodiodes (8) connected to

combined module (9) containing a photodiode and an IMU sensor.

Preferably, the photodiodes (8) can be mounted on the system board (4) at its edges equidistant. There may be four.

A module (9) with an IMU sensor and a light sensor is placed on the wrist of the glove (1). An additional advantage of the virtual glove (1) is the use of the shoulder module (10), which is placed on the shoulder (biceps), and connected to the system board with a wire (11) through the module (9), which is performed with a connecting connector (not shown in the drawings ).

An IMU sensor can be installed inside the shoulder module (10).

This mutual arrangement of IMU sensors using a sensor,

fixed in the shoulder module (10), allows you to effectively deal with the problem of "drift" data on the position of fingers and hand in space and more effectively track the movement of the shoulder joint. Unlike systems with external tracking (based on photodiodes or LEDs and external cameras), this solution can be used without any external devices, which is convenient for working with mobile devices (smartphones).

The shoulder module (10) can also be implemented in the wireless version, and connect to the computing module via radio (Bluetooth or Wifi). In this case, a battery and a radio module (not shown in the drawings) and a wire (11) are not used in a single module with a sensor (10).

Also, the shoulder module (10) can be optionally implemented not with the help of an IMU sensor, but with the help of one or several strain gauges (bend sensors). In this case, a microcontroller is installed in the shoulder module (10) to which the resistance strain gages are connected, and through them aggregates the information and transfers it to the main computing module of the motherboard (4).

The device works as follows.

The claimed solution allows you to receive and transfer data on the position of fingers, hands, elbow and shoulder joints to a computer or other device, and also implement tactile feedback by transmitting vibrations to the fingers. This is achieved as follows.

Gyroscopic / inertial sensors (IMU-sensors) are mounted on fingers: on the index, middle, ring fingers and little finger one sensor is installed in the penultimate phalanx of the finger, on the outside of the palm; two sensors are installed on the thumb - on the first and second phalanx, from the outside. Additionally, similar sensors are mounted on the hand, from the outside, and on the wrist, from the outside or inside of the palm.

The sensors can be fixed in these positions with a rag glove, sewn into the required positions, or each sensor can be attached independently with a rag (or other material) ring (fastening) worn on the finger / hand / wrist or any other in the way

providing immobility of the sensor relative to the finger / hand / wrist, respectively.

As an IMU-sensor can be used, for example:

https://www.digikey.com/product-detail/en/invensense/MPU-6000/1428-1005-l-

ND / 4038006

Sensors are connected via cable to the control module located on the outside of the palm in a plastic case. The control module contains microcontroller-based control electronics, a wireless radio module, and a battery.

Sensors constantly collect motion data and transmit it to the microcontroller, which transmits this data over the air to a computer or other device. A software driver is installed on the computer (or other device) that uses the data from the sensors (angular velocities and

acceleration) and converts them into quaternions turning space for the following joints: turning (bending) the last phalanx of the finger relative to the palm for

index, middle, ring fingers and little finger, in the vertical plane relative to the palm; rotation of the index, middle, ring fingers and little finger in the horizontal plane (palm plane); twist / fold of the thumb relative to the palm in space; rotate / fold the hand (palm) relatively elbow joint; rotation of the elbow joint relative to the shoulder joint.

Then, from the angles obtained, using the inverse kinematics algorithms (data on the joints of the joints and their linear dimensions), the turns of all the other joints and the relative position of all the joints of the hand are calculated.

The obtained data is then provided for access as a software application interface (API) and can be used by third-party

software manufacturers for any purpose.

Additionally, on the inside of the palm, on the penultimate phalanx of each finger, a vibration motor is placed, which is mounted in the same way as using IMU sensors, and also connected to the motherboard and microcontroller using a cable.

The application programming interface (API) allows the vibration motor to be launched independently on each of the fingers, setting the required parameters of rotary modulation (vibration frequency / intensity).

The command from the software interface is processed by the device driver and, via the radio channel, is transmitted to the microcontroller, which directly controls the vibration motor.

Thus, the vibration can be controlled by third-party software manufacturers using the application programming interface (API). As a vibration motor can be used, for example:

https://ru.aliexpress.com/item/50pcs-DC3V-0820-8-2-0mm-Mobile-phone-micro-flat- vibration-motor-Coin-motor-Mini-vibrator / 32783671636.html

The disadvantage of using IMU-sensors without additional external "tracking" (tracking - determining the location of moving objects over time using the camera) is the problem of "drift" - the accumulation of errors

gyroscopic / inertial sensors, a significant deviation of the received data from the real and, as a consequence, the impossibility of the exact absolute

positioning.

To solve this problem, it is necessary to impose restrictions on the data obtained, thereby limiting the mobility of the "virtual hand" - the hand is fixed in the elbow joint. Thus, all movements are interpreted within hemispheres that the hand can describe when the elbow joint is fixed - movements of the hand in the horizontal plane are completely ignored, which causes inconvenience in practice.

For this reason, one more gyroscopic / inertial sensor (IMU sensor) can be additionally attached to the shoulder joint, for example, inside the shoulder module (10) (Figure 4) and connected to the system board (4) using a cable (11) through socket module (9).

The data from the sensor is transmitted to the microcontroller, and then through the radio channel to the driver software, where they are used to calculate the rotation of the elbow joint relative to the shoulder joint, as well as the rotation of the elbow joint in a plane perpendicular to it. This allows you to more accurately calculate the position of the brush in space and give additional degrees of freedom to the hand.

Thanks to this approach, it is possible to fix the virtual model of the hand not in the elbow joint, but in the shoulder and with high accuracy to recognize and transfer to the computer (or other device) all possible hand movements, including in the horizontal plane.

To completely solve the problem of accumulated error in the calculated absolute position of the sensors, support for external "tracking" is used. For this purpose, two or more photodiodes reacting to flashes of light are placed on the control module. In this case, the housing for the control module is made of

transparent material that transmits light, either entirely, or only in those areas where the photodiodes are located. Another photodiode is located next to the IMU-sensor on the wrist. Optionally, another photodiode is also located along with the IMU-sensor on the shoulder joint.

Photodiodes register light signals sent by a pair of special external light emitters installed stationary, stationary and separate from

described controller (motherboard and a set of sensors in the form of gloves or other form), on opposite sides of it.

Emitters are programmed in a special way and generate light pulses of a certain length, shape and direction, at certain time intervals. Light can be emitted both in the visible and in the invisible (infrared, ultraviolet) ranges.

For example, the emitter can generate a set of short pulses of light, and then generate a narrow strip with a laser mounted on a rotating mechanism, rotating in a horizontal plane. Another emitter can do the same in parallel (but at different time intervals), but in the vertical plane.

The time intervals, the sequence of light pulses, the shape and nature of the pulses (for example, the laser light beam shape and the speed of the turning mechanism) are known in advance, these data are compared by the software driver with the photodiode registration time of various light pulses, thereby calculating their position relative to the emitters, and so image - the absolute position of the photodiodes in space. This data is then used to

adjusting the absolute position of the hand in space, calculated using IMU-sensors, and eliminating the accumulated error.

As a photodiode can be used, for example:

https://www.digikey.com/product-detail/en/osram-opto-semiconductors-inc/BPW-34-S- Z / 475-2659-1-ND / 1893861

As an external radiator can be used, for example:

https://www.microsoftstore.com/store/msusa/en_US/pdp/HTC-Vive-Base-

Station / productlD.5073718900

The claimed solution can be used to animate 3D models of a human (or other) hand in computer programs, as a way to interact with

interfaces in virtual (VR - virtual reality) or augmented (AR - augmented reality) realities, as well as video games, simulators of various types of activities and in any other tasks that require receiving, processing, storing or transmitting precise hand movements in space.

For example, the claimed solution can also be used in medicine, for patients with impaired motility of the hands to track the movements of the hands and fingers and stimulate their activity.

Feedback using vibration motors can be used to simulate the sensations of touching objects in virtual reality, when position data Hands and fingers are used to identify collisions with virtual objects and, if they are detected, apply vibration fingers to the corresponding contact point. The degree of vibration may depend on the shape of the contact - the size of the intersection area, or on the characteristics of the virtual object. Another possible scenario of using vibration is to confirm an event with interactions with virtual interfaces — for example, pressing a virtual button.

The claimed solution offers a new principle of placement of IMU-sensors and, as a result, high accuracy of the results: the relative position of the sensors allows you to get the necessary angles between the fingers, hand, elbow and shoulder joints; secondly, the location of the sensors and

Vibration motors allows you to release the last phalanx of fingers, which is convenient in practical application - the user can use touch screens, perform actions requiring fine motor skills of fingers, etc.

The advantage of using a combined approach (using IMU sensors and external tracking in one technology) is the high accuracy of measuring spatial displacement, even higher than any

alternative methods such as strain gauges (fold sensors) or solutions based only on IMU sensors. In addition, unlike methods based on external optical or other external tracking, this approach completely lacks "blind spots", even if there is no direct view between the LED (photodiode) and the camera (light emitter), the stated solution will still allow quite accurately calculate the position of the arm in space.

Claims

FORMULA

1. A virtual reality glove containing sensors located on the fingers of gloves, moreover, the sensors are connected to the motherboard, characterized in that the sensors used are IMU sensors located inside the glove made of fabric, each of the IMU sensors contains an accelerometer, gyroscope and magnetometer, with four IMU sensors located on the penultimate phalanges of the little finger, ring, middle and index fingers, two IMU sensors located on the first and second phalanges of the thumb, one IMU sensor located on the system board; on the fingers of the gloves are mounted vibration motors, one on each finger, and on all fingers except the big one, the vibration motors are fixed on the penultimate phalanges of the fingers, and on the thumb a vibration motor is fixed on the last phalanx; the glove is fixed to the motherboard where

computational module; IMU sensors, vibration motors, battery connected to the motherboard through wires; Combined module with IMU sensor and light sensor located on the wrist of the glove.

2. Glove according to claim 1, characterized in that the motherboard and the battery can be made in a single package of translucent plastic.

3. Glove according to claim 1, characterized in that photodiodes are mounted on the motherboard, connected to the combined module containing the photodiode and the IMU sensor.

4. Glove according to claim 1, characterized in that the photodiodes are installed on the system board at its edges equidistant.

5. Glove according to claim 1, characterized in that the system board has four photodiodes.

6. Glove according to claim 1, characterized in that, in addition, a shoulder module is mounted on the shoulder and connected to the system board by means of a wire through a combined module, which is made containing a connecting connector.

7. Glove according to claim 7, characterized in that an IMU sensor is installed inside the shoulder module.

8. The glove according to claim 7 or claim 8, characterized in that the shoulder module is made containing a wireless communication module and is configured to connect with system board over the air, and in the shoulder module is also a battery.

9. The glove according to claim 7, characterized in that the shoulder module contains a microcontroller and strain gages connected to it.