RU179301U1 - VIRTUAL REALITY GLOVE - Google Patents

Abstract

The utility model relates to manipulators, namely to virtual gloves designed to work with interactive devices, computer robotics, and a computer. The technical result of the utility model is the ability to receive and transmit data on the position of fingers, hands, elbow and shoulder joints to a computer or other device, as well as provide tactile feedback by transmitting vibration to the fingers. Also, the technical result is the ability to fix the virtual arm model not in the elbow joint, but in the shoulder and with high accuracy to recognize and transmit to the computer or other device all possible hand movements, including in the horizontal plane. Another advantage is the possibility of using it outside of stationary equipped rooms, which is convenient when using the solution together with mobile devices (smartphones) or portable virtual and augmented reality solutions. This technical result is achieved due to the fact that the claimed virtual reality glove containing sensors located on the fingers of the glove, and the sensors are connected to the system board, characterized in that the sensors used are IMU sensors located inside the glove made of fabric, each of IMU sensors contains an accelerometer, gyroscope and magnetometer, while four IMU sensors are located on the penultimate phalanges of the little finger, ring, middle and index fingers, two IMU sensors Position the first and second phalanges of the thumb, one IMU sensor located on the system board; vibration fingers are fixed on the fingers of the glove, one on each finger, and on all fingers except the thumb, vibration motors are mounted on the penultimate phalanges of the fingers, and on the thumb, the vibration motor is mounted on the last phalanx; the system board where the computing module is located is fixed on the glove; IMU sensors, vibration motors, battery connected to the system board via wires; combined module with IMU sensor and light sensor located on the wrist of the glove.

Description

The utility model relates to manipulators, namely to virtual gloves designed to work with interactive devices, computer robotics, and a computer.

The prior art knows the solution KR 100221335, which describes a system and method for transmitting with a glove the value of a sensor in a virtual reality system, including a series of sensors that measure the movements of the wrist and fingers and output an electrical signal; the multiplexer selects the number of sensor signals included in one of the sensor groups. The purpose of the patent is fundamentally different from the claimed solution. In analogs, it is precisely the high-speed data transfer between the sensors and the virtual reality system that is described. In the claimed solution, the data transfer method does not matter.

CN 1480822 describes a device for entering data for people with disabilities, which is a glove having many sensors capable of detecting finger movements much faster than is usually required to press keys in a regular keyboard; The sensor contains a virtual key detector and a key decoder, both of which are individually calibrated. The purpose of the glove and the method of detection are different. In the analogue, it is proposed to carry out interaction by touching certain sensors of the glove for people with disabilities, the touch will be transmitted to an external device. The solution we claimed determines precisely the movements and positions of the fingers and hands for transmission to an external device. In the application US 20150002401 described alternative gloves on the basis of the "key" for mobile devices, including at least one pair of gloves, many sensors located on the first side, at least two screens located on each side of the glove. The task of the glove is to enter some keys for mobile devices. In this case, the glove does not detect the movement and position of the fingers and hands, as the stated decision does.

US Pat. No. 6,870,526 describes a virtual mouse tracking mouse glove for controlling cursor movement and providing the function of a computer mouse. The presented glove is intended only for cursor and mouse control.

Our solution can not only control the mouse cursor, but also determine the movement of all fingers and hands to create a digital 3D model of the hand in virtual space.

US Pat. No. 7205979 describes a device for generating control signals for manipulating virtual objects in a computer system in accordance with gestures of an operator or other parts of the body; the device includes a glove, which includes sensors for detecting hand gestures, as well as hand position sensors connected to the glove and connected to a computer system to determine the position of the hand in relation to the system.

It differs in the principle of determining hand gestures. In the analogue, gestures are determined by means of external sensors installed, for example, on a computer monitor. Our solution uses inertial sensors built right into the glove.

Decision EP 2624238 describes a virtual mock-up with a tactile hand, having simulated objects that are manipulated with hand-worn gloves while the machine is operating. In the glove on the fingers are 3D common motion detection tools integrated with at least one interaction task sensor and one tactile component. The analogue is not specified how motion sensors are used. In the claimed solution, the use of inertial sensors with an accelerometer and a gyroscope is specified.

WO 98050839, describes a gesture-based data management system for processing real-time computer animations, such as virtual reality, and is computer-controlled. The system includes a digital glove for managing data based on operator’s hand gestures. The solution is based on the use of materials that change their resistance to bending.

The difference of the claimed utility model is the use of inertial sensors. In the international application WO 2007129663 (publ.: 15.11.2007), an input device is described using sensors mounted on a finger. The user wears gloves with fingers, each of which has a motion detection sensor and a tactile sensor for each of the fingertips on both hands. Motion information at each of the fingertips is transmitted to various information devices, such as a personal computer (PC), handheld computer, and cell phone, so that the positions of the corresponding fingertips are displayed on the virtual keyboard on the display of one of the information processing devices.

The analogue is not specified how motion sensors are used. In the claimed solution, the use of inertial sensors with an accelerometer and a gyroscope is specified.

The closest analogue is a virtual glove according to the patent US 9060385, publ.: 06.16.2015. The prototype describes a virtual reality glove containing sensors located on the fingers of the glove and position sensors of hands located on the phalanges of the fingers of the glove, the sensors being connected to a microprocessor. The disadvantage of the prototype and other known solutions is the following. As you know, the tactile analyzer has a high ability for spatial localization. Its characteristic feature is the rapid development of adaptation (addiction), i.e. the disappearance of a feeling of touch or pressure. The adaptation time depends on the strength of the stimulus; for different parts of the body, it ranges from 2 to 20 seconds. Thanks to adaptation, we do not feel the touch of clothing on the body. See [Ekzertseva Ekaterina Vadimovna, Topic 1.1. General life safety issues, Russian State Technological University named after K.E. Tsiolkovsky (MATI), Lectures, http://www.studfiles.ru/preview/854779/page:6/] Similar problems with working with virtual gloves are caused by well-known analogues, which reduces the user’s perception effect over a period of time working in them .

In addition, the technical problem of the known solutions when using IMU sensors without additional external “tracking” is the problem of “drift” - accumulation of the error of gyroscopic / inertial sensors, a significant deviation of the received data from real ones and, as a result, the impossibility of accurate absolute positioning.

The objective of the utility model is to resolve these problems.

The technical result of the utility model is the ability to receive and transmit data on the position of fingers, hands, elbow and shoulder joints to a computer or other device, as well as provide tactile feedback by transmitting vibration to the fingers.

Also, the technical result is the ability to fix the virtual arm model not in the elbow joint, but in the shoulder and with high accuracy to recognize and transmit to the computer or other device all possible hand movements, including in the horizontal plane.

Another advantage over other well-known devices based only on external tracking is the possibility of using them outside of stationary equipped rooms, which is convenient when using the solution together with mobile devices (smartphones) or portable solutions of virtual and augmented realities.

This technical result is achieved due to the fact that the claimed virtual reality glove containing sensors located on the fingers of the glove, and the sensors are connected to the system board, characterized in that the sensors used are IMU sensors located inside the glove made of fabric, each of IMU sensors contains an accelerometer, gyroscope and magnetometer, while four IMU sensors are located on the penultimate phalanges of the little finger, ring, middle and index fingers, two IMU sensors Position the first and second phalanges of the thumb, one IMU sensor located on the system board; vibration fingers are fixed on the fingers of the glove, one on each finger, and on all fingers except the thumb, vibration motors are mounted on the penultimate phalanges of the fingers, and on the thumb, the vibration motor is mounted on the last phalanx; the system board where the computing module is located is fixed on the glove; IMU sensors, vibration motors, battery connected to the system board via wires; combined module with IMU sensor and light sensor located on the wrist of the glove.

If necessary, the motherboard and battery can be made in a single housing made of translucent plastic.

Photodiodes are installed on the system board and connected to a combined module containing a photodiode and an IMU sensor.

Preferably, the photodiodes are mounted on the system board along its edges equidistant.

Preferably, four photodiodes are installed on the system board.

An additional shoulder module can be mounted on the shoulder, and connected to the system board using a wire through a combined module, which is made containing a connecting connector.

An IMU sensor can be installed inside the shoulder module.

The shoulder module may be configured to include a wireless module and configured to connect to the system board via a radio channel, wherein a battery is also located in the shoulder module.

The shoulder module, as an option, contains a microcontroller and strain gauges connected to it.

Brief Description of the Drawings

In FIG. 1 shows a diagram of a glove device using on the wrist and IMU sensor and without using photodiodes.

In FIG. 2 shows a diagram of a glove device using photodiodes and a combined module located on the wrist containing a photodiode and an IMU sensor.

In FIG. Figure 3 shows a diagram of a glove device without using photodiodes in the system board and located on the wrist of a combined module containing a photodiode and an IMU sensor.

In FIG. 4 shows a diagram of a glove module connecting to a glove.

In the drawings: 1 - a glove, 2 - an inertial sensor (sensor) IMU, combining an accelerometer, gyroscope and magnetometer, 3 - a vibration motor designed to transmit sensations of vibration, 4 - system board, 5 - battery, 6 - wires, 7 - a case combining the system board and the battery, 8 - a photodiode, 9 - a combined module, 10 - a shoulder module, 11 - a connecting wire between the combined module and the shoulder module.

Utility Model Implementation

The utility model (Fig. 2) consists of a glove (1) made of fabric, inside of which are located: IMU sensors (2), each of which contains an accelerometer, gyroscope and magnetometer, while four IMU sensors (2) are located on penultimate phalanges of the little finger, ring, middle and index fingers, two IMU sensors (2) are located on the first and second phalanxes of the thumb, one IMU sensor (2) is located on the system board.

On the fingers of the glove (1), vibration motors (3) are also fixed, one on each finger, and on all fingers except the thumb, vibration motors (3) are mounted on the penultimate phalanges of the fingers, and on the thumb, the vibration motor (3) is mounted on the last phalanx.

On the glove (1), the motherboard (4) is fixed, where the computing module is located. In this case, the Bluetooth wireless module, for communication with a computer or other device over the air, is installed on the system board (4).

The system board (4) is mounted on top of the battery (5), which is attached to the glove (1).

IMU sensors (2), vibration motors (3), battery (5) are connected to the system board (4) via wires (6).

If necessary, the system board (4) and the battery (5) can be made in a single housing (7) made of translucent plastic.

Photodiodes (8) are also installed on the system board (4), connected to a combined module (9) containing a photodiode and an IMU sensor.

Preferably, the photodiodes (8) can be mounted on the system board (4) at its edges equidistant. There may be four.

A module (9) with an IMU sensor and a light sensor is located on the wrist of the glove (1). An additional advantage of the virtual glove (1) is the use of the shoulder module (10), which is located on the shoulder (biceps), and connected to the system board using the wire (11) through the module (9), which is made containing a connecting connector (not shown in the drawings ) An IMU sensor can be installed inside the shoulder module (10).

This mutual arrangement of IMU sensors using a sensor mounted in the shoulder module (10), allows you to effectively deal with the problem of "drift" data on the position of the fingers and hands in space and more effectively track the movement of the shoulder joint. Unlike systems with external tracking (based on photodiodes or LEDs and external cameras), this solution can be used without any external devices, which is convenient for working with mobile devices (smartphones).

The shoulder module (10) can also be implemented wirelessly, and connected to the computing module via a radio channel (Bluetooth or Wifi). In this case, a battery and a radio module (not shown in the drawings) are located in a single module with a transmitter (10) and the wire (11) is not used.

Also, the shoulder module (10) can be optionally implemented not using an IMU sensor, but using one or more strain gauges (bend sensors). In this case, a microcontroller is installed in the shoulder module (10), to which strain gages are connected, and information is aggregated through them and transferred to the main computing module of the system board (4).

The utility model works as follows.

The claimed solution allows you to receive and transmit to a computer or other device data on the position of the fingers, hands, elbows and shoulder joints, as well as provide tactile feedback by transmitting vibration to the fingers. This is achieved as follows.

Gyroscopic / inertial sensors (IMU sensors) are installed on the fingers: on the index, middle, ring fingers and little fingers, one sensor is installed in the penultimate phalanx of the finger, on the outside of the palm; two sensors are installed on the thumb - on the first and second phalanx, from the outside. Additionally, similar sensors are installed on the hand, on the outside, and on the wrist, on the outside or inside of the palm.

The sensors can be fixed in these positions with a rag glove, being sewn in the required positions, or each sensor can be fixed independently using a rag (or from another material) ring (mount), worn on the finger / wrist / wrist, or any other in a way that ensures the immobility of the sensor relative to the finger / hand / wrist, respectively. As an IMU sensor, it can be used, for example: https://www.digikey.com/product-detail/en/invensense/MPU-6000/1428-1005-1-ND/4038006 The sensors are connected to the control module using a cable, located on the outside of the palm in a plastic case. The control module contains control electronics based on a microcontroller, a wireless radio module, and a battery.

Sensors constantly collect motion data and transmit them to the microcontroller, which transmits this data via radio channel to a computer or other device. A driver software is installed on a computer (or other device) that uses data from sensors (angular velocities and acceleration vectors) and converts them into quaternions of space rotation for the following joints: rotation (bend) of the last phalanx of the finger relative to the palm for the index, middle, ring fingers and little finger, in a vertical plane relative to the palm; rotation of the index, middle, ring fingers and little finger in the horizontal plane (palm plane); rotation / folding of the thumb relative to the palm in space; turn / bend of the hand (palm) relative to the elbow joint; rotation of the elbow joint relative to the shoulder joint. Then, from the obtained angles, using the inverse kinematics algorithms (data on the joints of the joints and their linear dimensions), the turns of all other joints and the relative position of all joints of the wrist are calculated.

The data obtained is then provided for access in the form of a software application interface (API) and can be used by third-party software manufacturers for any purpose.

Additionally, on the inner side of the palm, on the penultimate phalanx of each finger, a vibration motor is mounted, which is mounted in a similar way to IMU sensors, and is also connected via cable to the system board and microcontroller.

The application program interface (API) allows you to start the vibration motor independently on each of the fingers, setting the necessary parameters for pulse-width modulation (frequency / intensity of vibration).

The command from the software interface is processed by the device driver and, through the radio channel, is transmitted to the microcontroller, which controls the vibration motor directly.

In this way, vibration can be controlled by third-party software manufacturers using the application programming interface (API). As a vibration motor, it can be used, for example: https://ru.aliexpress.com/item/50pcs-DC3V-0820-8-2-0mm-Mobile-phone-micro-flat-vibration-motor-Coin-motor-Mini -vibrator / 32783671636.html

The disadvantage of using IMU sensors without additional external "tracking" (tracking - determining the location of moving objects in time using the camera) is the problem of "drift" - the accumulation of inaccuracy of gyroscopic / inertial sensors, a significant deviation of the received data from real and, as a result, the impossibility of accurate absolute positioning.

To solve this problem, it is necessary to impose restrictions on the data obtained, thereby limiting the mobility of the "virtual arm" - the arm is fixed in the elbow joint. Thus, all movements are interpreted within the hemisphere, which the brush can describe when the elbow is fixed - arm movements in the horizontal plane are completely ignored, which is inconvenient in practice.

For this reason, another gyroscopic / inertial sensor (IMU sensor) can be additionally fixed on the shoulder joint, for example, inside the shoulder module (10) (Fig. 4) and connected to the system board (4) using a cable (11) through combined module connector (9).

Data from the sensor is transmitted to the microcontroller, and then through the radio channel to the driver software, where it is used to calculate the rotation of the elbow joint relative to the shoulder, as well as the rotation of the elbow joint in a plane perpendicular to it. This allows you to more accurately calculate the position of the brush in space and give additional degrees of freedom to the hand.

Thanks to this approach, it is possible to fix the virtual arm model not in the elbow joint, but in the shoulder and with high accuracy to recognize and transmit to the computer (or other device) all possible hand movements, including in the horizontal plane.

To completely solve the problem of accumulated error in the calculated absolute position of the sensors, support for external "tracking" is used. For this, two or more photodiodes that respond to flashes of light are placed on the control module. In this case, the housing for the control module is made of a transparent material that transmits light, either in its entirety, or only in those areas where the photodiodes are located. Another photodiode is located next to the IMU sensor on the wrist. Optionally, another photodiode is also located with the IMU sensor on the shoulder joint.

Photodiodes register light signals sent by a pair of special external light emitters installed stationary, motionless and separately from the described controller (system board and a set of sensors in the form of a glove or other form), on different sides of it.

The emitters are programmed in a special way and generate light pulses of a certain length, shape and direction, at certain intervals. Light can be emitted both in the visible and in the invisible (infrared, ultraviolet) ranges.

For example, an emitter can generate a set of short pulses of light, and then generate a narrow band of laser mounted on a rotary mechanism rotating in a horizontal plane. Another radiator can do the same thing in parallel (but at different time intervals), but in a vertical plane. The time intervals, the sequence of light pulses, the shape and nature of the pulses (for example, the shape of the laser light beam and the speed of the rotary mechanism) are known in advance, these data are compared by the driver software with the time of registration of various light pulses by the photodiodes, so their position relative to the emitters is calculated, and image - the absolute position of the photodiodes in space. Then these data are used to correct the absolute position of the hand in space, calculated using IMU sensors, and to eliminate the accumulated error.

As a photodiode can be used, for example: https://www.digikey.com/product-detail/en/osram-opto-semiconductors-inc/BPW-34-S-Z/475-2659-1-ND/1893861

As an external emitter it can be used, for example: https://www.microsoftstore.com/store/msusa/en_US/pdp/HTC-Vive-Base-Station/productlD.5073718900

The claimed solution can be used to animate 3D models of a human (or other) hand in computer programs, as a way of interacting with interfaces in virtual (VR - virtual reality) or augmented (AR - augmented reality) realities, as well as video games, simulators of various activities and in any other tasks that require the receipt, processing, storage or transmission of precise hand movements in space.

For example, the claimed solution can also be used in medicine for patients with impaired motor skills to monitor the movements of the hands and fingers and stimulate their activity.

Feedback using vibromotors can be used to simulate the sensations of touching objects in virtual reality, when data on the position of the hand and fingers are used to determine collisions with virtual objects and, if they are detected, to apply vibration fingers to the contact point. Moreover, the degree of vibration may depend on the form of contact - the size of the intersection area, or on the characteristics of the virtual object. Another possible scenario for using vibration is to confirm an event by interacting with virtual interfaces - for example, clicking on a virtual button.

The claimed solution offers a new principle for the placement of IMU sensors and, as a result, high accuracy of the results: the relative position of the sensors allows you to get the necessary angles between the fingers, hand, elbow and shoulder joints; secondly, this arrangement of sensors and vibration motors allows you to release the last phalanx of the fingers, which is convenient in practical use - the user can use touch screens, perform actions requiring fine motor skills of the fingers, etc.

The advantage of using a combined approach (using IMU sensors and external tracking in one technology) is the high accuracy of spatial displacement measurements, even higher than any alternative methods, such as strain gages (bend sensors) or solutions based only on IMU sensors. In addition, unlike methods based on external optical or other external tracking, this approach completely lacks “blind spots”, even if there is no direct view between the LED (photodiode) and the camera (light emitter), the claimed solution will still allow for a fairly accurate calculate the position of the hand in space.

Another advantage over other well-known devices based only on external tracking is the possibility of using them outside of stationary equipped rooms, which is convenient when using the solution together with mobile devices (smartphones) or portable solutions of virtual and augmented realities.

Claims (9)

1. A virtual reality glove comprising sensors located on the fingers of a glove, the sensors being connected to a system board, characterized in that the sensors used are IMU sensors located inside a glove made of fabric, each IMU sensor containing an accelerometer, a gyroscope and magnetometer, while four IMU sensors are located on the penultimate phalanxes of the little finger, ring, middle and index fingers, two IMU sensors are located on the first and second phalanxes of the thumb, one IMU sensor located on the system board; vibration fingers are fixed on the fingers of the glove, one on each finger, and on all fingers except the thumb, vibration motors are mounted on the penultimate phalanges of the fingers, and on the thumb, the vibration motor is mounted on the last phalanx; the system board where the computing module is located is fixed on the glove; IMU sensors, vibration motors, battery connected to the system board via wires; combined module with IMU sensor and light sensor located on the wrist of the glove. 2. The glove according to claim 1, characterized in that the motherboard and the battery can be made in a single housing made of translucent plastic. 3. The glove according to claim 1, characterized in that photodiodes are installed on the system board and connected to a combined module containing a photodiode and an IMU sensor. 4. The glove according to claim 1, characterized in that the photodiodes are mounted on the system board along its edges equidistant. 5. The glove according to claim 1, characterized in that four photodiodes are installed on the system board. 6. The glove according to claim 1, characterized in that the shoulder module is additionally mounted on the shoulder and connected to the system board using a wire through a combined module, which is made containing a connecting connector. 7. The glove according to claim 7, characterized in that an IMU sensor is installed inside the shoulder module. 8. The glove according to claim 7 or 8, characterized in that the shoulder module is made comprising a wireless communication module and is configured to connect to the system board via a radio channel, wherein a battery is also located in the shoulder module. 9. The glove according to claim 7, characterized in that the shoulder module contains a microcontroller and strain gauges connected to it.

Images