WO2023246305A1

WO2023246305A1 - Whole-body posture tracking and haptic device and virtual reality system

Info

Publication number: WO2023246305A1
Application number: PCT/CN2023/091411
Authority: WO
Inventors: 杜伟华; 张�浩; 陈丽莉; 韩鹏; 何惠东; 石娟娟; 秦瑞峰; 姜倩文; 赵砚秋
Original assignee: 京东方科技集团股份有限公司; 北京京东方显示技术有限公司
Priority date: 2022-06-21
Filing date: 2023-04-28
Publication date: 2023-12-28
Also published as: CN115202471A

Abstract

The present disclosure belongs to the technical field of virtual reality (VR) augmented display, and provides a whole-body posture tracking and haptic device and a virtual reality system. The whole-body posture tracking and haptic device of the present disclosure comprises: a body structure, at least one action detection unit, a control unit, and at least one haptic control unit. The body structure is configured to be worn on a limb of a user. The action detection unit is mounted on the body structure, and is configured to detect a limb action of the user and generate motion posture information of the user. The control unit is configured to process received motion posture information, generate a first control signal, and transmit same to a VR device, so that the VR device controls a limb action of a virtual user, and after receiving a touch signal which is fed back by the VR device and which indicates that the virtual user is touched, generates a second control signal according to the touch signal. The haptic control unit is configured to feed back real haptic sensing to the limb of the user by means of the body structure according to the second control signal.

Description

A whole-body posture tracking and tactile device and virtual reality system

Technical field

The present disclosure belongs to the field of virtual reality enhanced display technology, and specifically relates to a whole-body posture tracking and tactile device and a virtual reality system.

Background technique

Virtual reality technology (VR; Virtual Reality) is a new practical technology developed in the 20th century. Virtual reality technology includes computers, electronic information, and simulation technologies. Its basic implementation method is that computers simulate virtual environments to give people a sense of immersion in the environment. With the continuous development of social productivity and science and technology, the demand for VR technology in all walks of life is increasingly strong.

Most mobile VR headsets have rotational tracking (3DoF), so you can look up or down and tilt left or right. But if you try to tilt or move the position of your head, it won't be tracked. Virtual reality is currently only based on our eyes and ears.

Contents of the invention

The present invention aims to solve at least one of the technical problems existing in the prior art and provide a whole-body posture tracking and tactile device and a virtual reality system.

In a first aspect, embodiments of the present disclosure provide a whole-body posture tracking and haptic device, which includes: a main structure, at least one action detection unit, a control unit, and at least one tactile control unit; wherein,

The main structure is configured for the user to wear on the body;

The action detection unit is installed on the main structure and configured to detect the user's body movements and generate user movement posture information;

The control unit is configured to process the received motion posture information, generate a first control signal, and transmit it to the VR device, so that the VR device can control the body movements of the virtual user, and after receiving the After the VR device feeds back the touch signal that the virtual user is touched, a second control signal is generated according to the touch signal;

The tactile control unit is configured to feedback real tactile sensation to the user's limb through the main structure according to the second control signal.

In some examples, the motion posture information includes the rotation angle of the motion joint; the action detection unit includes a first detection module and a second detection module;

The first detection module is configured to detect the electromyographic signal generated by the user's skeletal muscles and process it;

The second detection module is configured to detect the user's skin surface tension signal and process it;

The control unit is configured to obtain the rotation angle of the user's moving joints based on the processed electromyographic signal and the skin surface tension signal and through a first preset algorithm, and obtain the rotation angle of the user's moving joints based on the processed Angle, through the second preset algorithm, the user's motion posture information is obtained to generate the first control signal.

In some examples, the first detection module includes a myoelectric signal electrode and a first bandpass amplifier; the second detection module includes a skin surface tension strain gauge and a second bandpass amplifier;

The electromyographic signal electrode is configured to detect the electromyographic signal generated by the user's skeletal muscles;

The first bandpass amplifier is configured to amplify the electromyographic signal and transmit it to the control unit;

The skin surface tension strain gauge is configured to detect the user's skin surface tension signal;

The second bandpass amplifier is configured to amplify the skin surface tension signal and transmit it to the control unit.

In some examples, the control unit includes an analog-to-digital conversion module, a first calculation module, and a first control module;

The analog-to-digital conversion module is configured to convert the processed electromyographic signal and the skin surface tension signal into a first digital signal and a second digital signal respectively;

The first calculation module is configured to obtain the rotation angle of the user's motion joint according to the first digital signal and the second digital signal and through a first preset algorithm, and through the second Preset algorithm to obtain the user's movement posture information;

The first control module is configured to generate a first control signal according to the user's motion posture information and transmit it to the VR device so that the VR device can control the body movements of the virtual user, and after receiving After the VR device feeds back a touch signal indicating that the virtual user has been touched, a second control signal is generated based on the touch signal.

In some examples, the first calculation module is specifically configured to obtain the rotation angle of the user's motion joint according to the first digital signal and the second digital signal and through a first preset algorithm, and according to The pre-established kinematic model calculates the spatial posture coordinates of the current position point of each moving joint relative to the initial point of each moving joint, thereby obtaining the user's movement posture information; wherein the initial point is the user's Angle information of each moving joint in the initial posture.

In some examples, the main structure is composed of a plurality of hollow wire tubes; the tactile control unit includes a first switch module, a second switch module and a touch module;

The first switch module and the second switch module are both connected to the hollow wire tube, and are used to control the entry and discharge of gas into the hollow wire tube according to the second control signal; and when the touch module is arranged on On the hollow wire tube defined by the first switch module and the second switch module, when the user wears the main structure, the touch module is located on the side of the hollow wire tube close to the human body.

In some examples, the control unit includes a second control module and a second computing module;

The second calculation module is configured to calculate the location information of the user actually being touched based on the touch signal of the virtual user being touched fed back by the VR device;

The second control module is configured to generate the second control signal according to the location information of the user's actual touch.

In some examples, the material of the hollow wire tube includes fiber material.

In some examples, the first and second switching modules include valves.

In some examples, the touch module includes a striking hammer.

In some examples, the haptic control unit further includes a suction component configured to inflate gas into the hollow wire tube and exhaust the gas in the hollow wire tube.

In some examples, the suction assembly is an air pump.

In a second aspect, embodiments of the present disclosure also provide a virtual reality system, which includes the above-mentioned whole-body posture tracking and haptic device and a VR device; the VR device is communicatively connected with the whole-body posture tracking and haptic device.

In some examples, the whole-body posture tracking and haptic device and the VR device are connected through WIFI or Bluetooth.

In some examples, the VR device includes a VR helmet.

Description of the drawings

Figure 1 is a schematic diagram of a human kinematics model;

Figure 2a is a rendering of the use of whole-body posture tracking and tactile equipment;

Figure 2b is a schematic diagram of the textile details of part I in Figure 2a;

Figure 3 is a schematic diagram of virtual reaction force in virtual touch;

Figure 4 is a schematic diagram of virtual pain in virtual touch;

Figure 5 is a flow chart of whole body posture tracking;

Figure 6 is a virtual tactile flow chart;

Figure 7 is a schematic diagram of human elbow joint posture tracking;

Figure 8 Action detection principle block diagram;

Figure 9 is a schematic diagram of the virtual reality system.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

Unless otherwise defined, technical terms or scientific terms used in this disclosure shall have the usual meaning understood by a person with ordinary skill in the art to which this disclosure belongs. "First", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Likewise, similar words such as "a", "an" or "the" do not indicate a quantitative limitation but rather indicate the presence of at least one. Words like "include" or "include" mean Elements or things appearing before the word include elements or things listed after the word and their equivalents, without excluding other elements or things. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right", etc. are only used to express relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

Existing virtual reality (VR; Virtual Reality) technology usually requires the use of VR equipment to close people's vision and hearing to the outside world, guiding users to create a feeling of being in a virtual environment. Common VR devices 2 currently on the market include VR glasses, VR helmets, etc.

Among them, VR helmets generally include lenses, displays, and cables; among them, the lens, as one of the most critical components, not only plays a role in linking the user's eyes and VR content, but also determines the user's visual effects to a large extent. ;

In addition, high-performance displays enable VR helmets to have enough pixel density to display clear images and make motion pictures in VR smooth and smooth; high-end VR helmets also use dual-screen displays to provide stereoscopic 3D effects, with each screen Each eye displays a slightly offset image, and our brains automatically "glue" them together into one image, creating an illusion of depth in the process;

In addition, VR helmets are also equipped with built-in sensors so that virtual characters can follow the user's head to make corresponding changes to obtain a more accurate display. However, the tracking methods of most VR devices 2 in the existing technology still revolve around the head and hands, which makes the interaction method in the virtual environment single and lacks the realism of virtual reality. Even if it can be tracked through a tracker or using a constellation of a specific shape on the tracking object, it cannot be achieved because such a system is complex and difficult to track detailed postures. At the same time, how to allow users to experience the touch of different body parts of the virtual user and integrate virtual touch into virtual reality to improve the real experience of virtual reality is still an important issue in this field.

In view of this, in the embodiment of the present disclosure, a whole-body posture tracking and haptic device 1 is provided, which detects the user's body movements through the movement detection unit 20 and generates user movement posture information, thereby realizing the user's whole-body posture tracking; through the main structure 10 and the tactile control unit 40 realize virtual tactile sensation, and the VR device 2 is responsible for general head position and posture tracking, thereby greatly improving the performance of virtual reality. The sense of real experience plays a certain role in promoting the development of true virtual reality in virtual reality systems.

The display module according to the embodiment of the present disclosure will be described below with reference to the accompanying drawings and specific embodiments.

In a first aspect, embodiments of the present disclosure provide a whole-body posture tracking and haptic device 1, which can be used to interact with information from a VR device 2 worn on the user's head. The whole body posture tracking and haptic device 1 includes: a main structure 10 , at least one action detection unit 20 , a control unit 30 , and at least one tactile control unit 40 . Among them, the main structure 10 is configured to be worn by the user on the limbs; the action detection unit 20 is installed on the main structure 10, and the action detection unit 20 is configured to detect the user's body movements and generate user motion posture information; the control unit 30 is Configured to process the received motion posture information, generate a first control signal, and transmit it to the VR device 2 so that the VR device 2 can control the body movements of the virtual user, and when the virtual user receives feedback from the VR device 2 After receiving the touch signal, a second control signal is generated according to the touch signal; the tactile control unit 40 is configured to feedback real tactile sensation to the user's limb through the main structure 10 according to the second control signal.

It should be noted that the main structure 10 in the embodiment of the present disclosure can be clothes. When the user uses the whole body posture tracking and tactile device 1, the user wears the main structure 10. At this time, the main structure 10 can be relatively fixed on the user's limbs. (such as tights) to fit the human skin. The motion detection unit 20 is fixed on the main structure 10. When the user wears the main structure 10, the position of the motion detection unit 20 generally corresponds to the user's skeletal muscles and is located on the side of the main structure 10 away from the user's skin.

In the embodiment of the present disclosure, since the whole body posture tracking and haptic device 1 includes a main structure 10 , at least one action detection unit 20 , a control unit 30 , and at least one tactile control unit 40 , the control unit 30 can detect the received action detection unit 20 The detected real motion posture information of the user is processed, a first control signal is generated, and transmitted to the VR device 2 so that the VR device 2 can control the body movements of the virtual user, and after receiving feedback from the VR device 2 After the virtual user is touched by the touch signal, a second control signal is generated according to the touch signal. The tactile control unit 40 can feedback real tactile sensation to the user's limbs through the main structure 10 according to the second control signal. Therefore, the present disclosure is used to implement The whole-body posture tracking and tactile device 1 of the example can realize the tracking of the user's whole-body posture, and can give the user virtual tactile perception according to the display of the VR device 2, thus greatly improving High virtual display real experience.

In some examples, the motion posture information may be the rotation angle of the motion joint. The action detection unit 20 includes a first detection module 21 and a second detection module 22 . Specifically, the first detection module 21 is configured to detect the electromyographic signal generated by the user's skeletal muscles and process it; the second detection module 22 is configured to detect the user's skin surface tension signal and process it; the control unit 30 is It is configured to obtain the rotation angle of the user's moving joints according to the processed electromyographic signal and the skin surface tension signal through the first preset algorithm, and obtain the rotation angle of the user's moving joints through the second preset algorithm. The user's motion posture information is used to generate the first control signal.

Further, the first detection module 21 includes a myoelectric signal electrode 211 and a first bandpass amplifier 212; the second detection module 22 includes a skin surface tension strain gauge 221 and a second bandpass amplifier 222; the myoelectric signal electrode 211 is configured In order to detect the electromyographic signal generated by the user's skeletal muscles; the first bandpass amplifier 212 is configured to amplify the electromyographic signal and transmit it to the control unit 30; the skin surface tension strain gauge 221 is configured to detect the user's skin surface tension signal; the second bandpass amplifier 222 is configured to amplify the skin surface tension signal and transmit it to the control unit 30 .

It should be noted that the action detection unit 20 is installed on the main structure 10, and its projection on the main structure 10 covers the outside of the skeletal muscles of the human body; the myoelectric signal electrodes 211 and the first bandpass amplifier 212 in the action detection unit 20 The number of skin surface tension strain gauges 221 and second bandpass amplifiers 222 can be adjusted according to different body parts.

Specifically, when the muscle contracts, the myoelectric signal electrode 211 detects the myoelectric current. The greater the degree of muscle contraction, the greater the myoelectric current generated. By detecting the size of the myoelectric current, the degree of muscle contraction can be inferred, and then the degree of muscle contraction can be calculated. Out of the joint rotation angle, the virtual user's movements can be controlled through the action angles of each joint of the human body, thereby completing the whole body posture tracking; the skin surface tension strain gauge 221 can detect the skin stress changes during muscle contraction and relaxation. When the muscle contracts, the skin Under the action of squeezing force, the skin surface tension strain gauge 221 is under pressure. When the muscles relax, the skin is under surface tension. The skin surface tension strain detects the tension, and the skin stress change current can be calculated more accurately by combining the myoelectric signal current. Out joint rotation angle.

It should be noted that the first bandpass amplifier 212 and the second bandpass amplifier 222 are used to filter and amplify the electromyographic signal and the skin surface tension signal, but there are more than just signal amplifiers used to amplify the signal. It is limited to a bandpass amplifier, so the type of signal amplifier is not limited here, that is, as long as the detected signal can be effectively amplified.

Further, the control unit 30 includes an analog-to-digital conversion module 31, a first calculation module and a first control module; the analog-to-digital conversion module 31 is configured to convert the processed electromyographic signal and skin surface tension signal into a first digital signal respectively. and a second digital signal. The first calculation module is configured to obtain the rotation angle of the user's motion joints based on the first digital signal and the second digital signal through the first preset algorithm, and obtain the user's motion posture information through the second preset algorithm. The first control module is configured to generate a first control signal according to the user's motion posture information and transmit it to the VR device 2 so that the VR device 2 can control the body movements of the virtual user, and after receiving feedback from the VR device 2 After the virtual user is touched by the touch signal, a second control signal is generated according to the touch signal.

In some examples, the first calculation module is specifically configured to obtain the rotation angle of the user's motion joint according to the first digital signal and the second digital signal and through the first preset algorithm, and according to the pre-established kinematic model, Calculate the spatial pose coordinates of the current position point of each moving joint relative to the initial point of each moving joint, thereby obtaining the user's movement posture information; where the initial point is the angle information of each movement joint of the user's initial posture.

Further, the first computing module includes a multimedia application processor (MAP; Multimedia Application Processor) and an inertial measurement unit (IMU; Inertial Measurement Unit). The first computing module is configured to first process the first digital signal and the second digital signal through the MAP. The digital signal is processed to obtain the rotation angle of the user's motion joints, and then the user's motion posture information is calculated through kinematic formulas based on the rotation angle of the user's motion joints.

Specifically, when using this system for the first time, virtual and real body calibration must be performed, that is, the physical size of the user's body, initial posture, and spatial coordinates of the end of the hands and feet must correspond to the virtual user. Among them, the physical size of the body is obtained by taking a full-body photo with the VR device 2; the initial posture is based on the initial posture of the user starting to use the disclosed system. After booting, the system will record the initial posture through full-body posture tracking. In the subsequent whole-body posture tracking process, Each joint angle of the initial posture is used as the initial point; the spatial coordinates of the hand and foot ends can be calculated from kinematics.

Establish the fixed point O ₁ and establish the X ₁ O ₁ Y ₁ coordinate system. The fixed point O ₁ is positioned relative to the VR device 2. unchanged, the relative posture is obtained and calculated by VR device 2 and IMU; take the position that the user needs to calculate as O ₂ and establish the X ₂ O ₂ Y ₂ coordinate system, and calculate X ₂ O ₂ Y ₂ relative to X ₁ O ₁ The coordinate transformation matrix of Y ₁ , by calculating the coordinates of the user's limbs relative to O ₁ , finally obtains the user posture.

Figure 1 is a schematic diagram of a human body kinematics model; the process for a user to control a virtual user through body movements is: the user makes corresponding actions according to the virtual screen, the whole body posture tracking and tactile device 1 performs posture tracking, and the control unit 30 sends the body posture coordinates to VR device 2. VR device 2 controls the actions of the virtual user through user gestures. Specifically, as shown in Figure 1, taking the left arm of the human body in the figure as an example, a kinematics model is established to calculate the relative spatial pose coordinates of point O ₂ with respect to point O ₁ ; the relative position of point O ₁ to VR device 2 is Fixed, the relative posture can be obtained and calculated by the IMU of the VR device 2 and the control unit 30. Now the joint angle information of the left arm of the human body in the figure can be obtained by tracking the whole body posture. Next, the position of point O ₂ relative to point O ₁ is calculated. Attitude coordinates:

In the above formula:

1) is the pose coordinate of θ ₂ relative to O ₁ ;

2) cθ ₂ is cosθ ₂ ;

3) sθ ₂ is sinθ ₂ ;

During the above calculation process, is the coordinate transformation matrix of the hand coordinate system X ₂ O ₂ Y ₂ relative to the central coordinate system X ₁ O ₁ Y ₁ of the control unit 30. By calculating the coordinates of the human limbs relative to O ₁ , the whole body posture information of the human body can be obtained. In this process, the user controls the virtual user through body movements. The user's process is: the user makes corresponding actions according to the virtual screen, the whole body posture tracking and haptic device 1 performs posture tracking, and the control unit 30 sends the user's posture coordinates to the VR device 2, thereby realizing the user's control of the virtual user's actions.

It can be understood that during the above calculation process, there are no strict requirements for the establishment of the hand coordinate system X ₂ O ₂ Y ₂ and the center coordinate system X ₁ O ₁ Y ₁ of the control unit 30 . Usually the control unit 30 is located at the center of gravity of the human body, which can make it more convenient to calculate the whole body posture information. However, the control unit 30 may not be located at the center of gravity of the human body, and it will not affect the tracking and calculation of the whole body posture at any position of the human body.

In some examples, after receiving the touch signal from the VR device 2 that the virtual user is touched, the control unit 30 generates a second control signal based on the touch signal. The tactile control unit 40 can pass the main body according to the second control signal. The structure 10 feeds back real tactile sensations to the user's limbs.

In some examples, the control unit 30 includes a second control module and a second calculation module; the second calculation module is configured to calculate the actual touch value of the user based on the touch signal fed back by the VR device 2 that the virtual user is touched. Position information; the second control module is configured to generate a second control signal according to the position information of the user's actual touch. The tactile control unit 40 can feedback real tactile sensation to the user's limbs through the main structure 10 according to the second control signal.

It should be noted that the tactile control unit 40 can exist independently of the control unit 30, that is, the control unit 30 does not directly control the implementation of virtual tactile sensation, but the tactile control unit 40 can also be controlled by the control unit 30, and the two are connected through communication to achieve Virtual haptic functionality.

Further, the tactile control unit 40 includes a first switch module 41, a second switch module 42 and a touch module 43; both the first switch module 41 and the second switch module 42 are connected to the hollow wire tube for controlling the operation according to the second control signal. , to control gas entering and exiting the hollow wire tube; and when the touch module 43 is disposed on the hollow wire tube defined by the first switch module 41 and the second switch module 42, and when the user wears the main structure 10, the touch module 43 is located The hollow wire tube is close to the side of the human body.

Specifically, the main structure 10 is composed of a plurality of hollow wire tubes woven; Figure 2b is a schematic diagram of the textile details of part I in Figure 2a. As shown in Figure 2b, the hollow wire tubes have a mesh-like staggered structure and the internal air pressure is adjustable. The flexibility of the hollow wire tube changes with the internal air pressure. The overall flexibility of the hollow wire tube can be changed by filling or discharging gas; the local air pressure of the hollow wire tube is controllable, and virtual touch is achieved by adjusting the local air pressure of the hollow wire tube.

In some examples, the material of the hollow wire tube includes fiber material, ie, a hollow fiber tube. Fiber materials have the characteristics of high strength, light weight, and good air permeability, so the main structure 10 made of hollow fiber tubes is suitable for human body wear. However, the material of the hollow wire tube is not limited to fiber materials. As long as the functional requirements of the main structure 10 are met, any type of material can be tried, but lightweight and high performance are preferred.

In some examples, the first switch module 41 and the second switch module 42 include, but are not limited to, valves. The first switch module 41 and the second switch module 42 are used to control the local pressure of the hollow line tube, so their forms are not limited.

In some examples, the touch module 43 includes a striking hammer; wherein, after the tactile control unit 40 receives the second control signal, the first switch module 41 and the second switch module 42 on both sides of the striking hammer are controlled to close, so that The local air pressure of the hollow wire tube becomes smaller, and the hammer hits the hitting point on the surface of the human skin under the action of pressure, thereby achieving virtual pain sensation.

It should be noted that instead of controlling the hammer to hit the human skin surface by turning off the first switch module 41 and the second switch module 42 on both sides of the hammer, the rise and fall of the hammer can be directly controlled to achieve virtual touch.

In some examples, the haptic control unit 40 further includes a suction assembly 44 configured to inflate gas into the hollow wire tube and exhaust the gas in the hollow wire tube. Virtual touch can be achieved by adjusting the air pressure inside the hollow wire tube.

In some examples, suction assembly 44 is an air pump. The air pump fills the hollow line pipe with gas and discharges the gas in the hollow line pipe. Specifically, the gas is introduced into the gas storage cylinder through the air guide tube, and then introduced into the hollow line tube. At the same time, the gas storage cylinder introduces the gas in the gas storage cylinder into the pressure regulating valve fixed on the air pump through an air guide tube, thereby controlling the air pressure in the gas storage cylinder. . When the air pressure in the air reservoir does not reach the pressure set by the pressure regulating valve, the gas entering the pressure regulating valve from the air reservoir cannot open the pressure regulating valve; when the air pressure in the air reservoir reaches the pressure set by the pressure regulating valve, , the gas entering the pressure regulating valve from the air storage tank opens the pressure regulating valve valve, enters the air channel in the air pump that communicates with the pressure regulating valve, and controls the air inlet of the air pump to be normally open through the air channel, so that the air pump runs without load. When the air pressure in the air storage tank is lower than the pressure set by the pressure regulating valve due to loss, the valve in the pressure regulating valve is returned by the return spring, disconnects the control air path of the air pump, and the air pump starts pumping air again. When out of use, the air pump will automatically purge the gas.

It should be noted that the form and quantity of the suction assembly 44 are not limited here in order to realize its functions of filling gas into the hollow wire tube and discharging the gas in the hollow wire tube. For example, a blower can also serve as the suction component 44. Of course, considering that the whole-body posture tracking and haptic device 1 provided by the present disclosure can be worn on the user's body, the weight and performance of the suction component 44 should be taken into consideration, that is, in the case of realizing the function of the suction component 44, the suction The lighter and more compact the suction assembly 44 is, the better.

In order to have a clearer understanding of the specific working principle of the whole-body posture tracking and haptic device 1 provided by the embodiment of the present disclosure, the following will be described with reference to specific examples.

Figure 2a is a rendering of whole body posture tracking and the use of tactile equipment; Figure 3 is a schematic diagram of virtual reaction force in virtual touch; Figure 4 is a schematic diagram of virtual pain in virtual touch; Figure 5 is a flow chart of whole body posture tracking; Figure 6 is a flow chart of virtual touch ; Referring to Figures 2a, 3, 4, 5, and 6, the action detection unit 20 of the whole body posture tracking and haptic device 1 is installed on the main structure 10 and is configured to detect the user's body movements. When the user moves, the action detection unit The unit 20 detects the user's body movements and generates user motion posture information and transmits it to the control unit 30. The control unit 30 is configured to process the received movement posture information, generate a first control signal, and transmit it to the VR device 2 for use. The VR device 2 controls the body movements of the virtual user to achieve full body posture tracking; when the user moves in the VR environment and is hit by a virtual character, a touch signal is generated and transmitted to the VR device 2, and the control unit 30 is in the VR device 2. After feeding back the touch signal that the virtual user is touched, a second control signal will be generated according to the touch signal. The tactile control unit 40 is configured to feedback real tactile sensation to the user's limbs through the main structure 10 according to the second control signal. This enables virtual touch.

Specifically, as shown in Figure 5, when using this system for the first time, virtual and real body calibration must be performed, that is, the physical size of the user's body, initial posture, and spatial coordinates of the end of the hands and feet must correspond to the virtual user. Among them, the physical size of the body is obtained by taking a full-body photo with the VR device 2; the initial posture is based on the initial posture of the user starting to use the disclosed system. After booting, the system will record the initial posture through full-body posture tracking. In the subsequent whole-body posture tracking process, Each joint angle of the initial posture is used as the initial point; the spatial coordinates of the hand and foot ends can be calculated from kinematics.

After the user performs virtual and real body calibration, whole-body posture tracking begins. Figure 7 is a schematic diagram of human elbow joint posture tracking; as shown in Figure 7, taking the tracking of the elbow joint as an example, on the outside of the triceps brachii and the outside of the biceps brachii The elbow joint motion detection units 20 are respectively attached. Figure 8 is a block diagram of action detection principle; such as As shown in FIG. 8 , each action detection unit 20 includes a first detection module 21 and a second detection module 22 . The first detection module 21 includes a myoelectric signal electrode 211 and a first bandpass amplifier 212 . The second detection module 22 It includes a skin surface tension strain gauge 221 and a second bandpass amplifier 222. When the muscle contracts, the myoelectric signal electrode 211 detects the myoelectric current. The greater the degree of muscle contraction, the greater the myoelectric current generated. By detecting the size of the myoelectric current, the degree of muscle contraction can be deduced, and then the joint rotation can be calculated. Angle, the movement angle of each joint of the human body can be used to control the movements of the virtual user, thereby completing the whole body posture tracking; the skin surface tension strain gauge 221 can detect the skin stress changes during muscle contraction and relaxation. When the muscle contracts, the skin is squeezed The skin surface tension strain gauge 221 is subject to pressure. When the muscles relax, the skin is subject to surface tension. The skin surface tension strain detects the tension. The skin stress change current and the electromyographic signal current can more accurately calculate the joint rotation. angle to complete whole-body posture tracking.

As shown in Figure 3, when the user tries to use the virtual user's arm to push an object in the virtual scene, the user waves his arm in the real environment, and the virtual user also pushes the object in the virtual scene according to the user's movements. In a real scene, when we push an object, we will feel the reaction force of the object on our hand or arm, and in this disclosure, we want to achieve this effect in the virtual scene. When the user waves his arm to push an object in the opposite direction as shown in Figure 3, the air pressure of the hollow wire tube on the outside of the arm becomes smaller, generating a force F1 that contracts toward the inside of the hollow wire tube. When the forearm is waving, Under the action of F1, the arm will feel a reaction force of F2, that is, the user also feels the reaction force of the virtual user at the same time, thereby realizing virtual touch.

As shown in Figure 6, when the virtual user is hit, the process of the user feeling pain is as follows: After the virtual user is hit, the VR device 2 will record the physical coordinates of the hit by the virtual user, and then transmit the position information to The tactile control unit 40 adjusts the pressure at the corresponding position of the user's body to produce a slight pain.

Specifically, as shown in Figure 4, when the user is playing a fighting game, the virtual opponent hits the virtual user. In order to make the virtual reality closer to the real scene, when the virtual user is hit, the user should also feel pain. . When the virtual user is hit somewhere on the body, corresponding to the hit point on the skin surface of the real user, at this time, the air pressure of the hollow line tubes on both sides of the first switch module 41 and the second switch module 42 of the hit point becomes smaller, and under the action of pressure down, the first switch module 41 and the second switch module 42 are subjected to F1 Under the action of the pulling force of F2, the touch module 43 hits the hitting point on the skin surface under the action of the pressure of F3, and the user feels a slight pain, thereby realizing virtual pain.

In a second aspect, embodiments of the present disclosure also provide a virtual reality system. Figure 9 is a schematic diagram of the virtual reality system; as shown in Figure 9, it includes the above-mentioned whole body posture tracking and haptic device 1 and VR device 2; VR device 2 is communicatively connected with the whole body posture tracking and haptic device 1.

In order to have a clearer understanding of the specific working principles of the virtual reality system provided by the embodiments of the present disclosure, the following is explained with reference to specific examples.

In one example, referring to Figure 9, the virtual reality system provided by the present disclosure includes a whole-body posture tracking and haptic device 1 and a VR device 2. The whole-body posture tracking and haptic device 1 mainly completes the whole-body posture tracking and virtual haptic functions, and the VR device 2 Mainly completes general head six-degree-of-freedom tracking. Whole body posture tracking is mainly completed by the action detection unit 20 arranged on the outer surface of the skeletal muscles of the whole body. The action detection unit 20 is mainly composed of myoelectric signal electrodes 211 and skin surface tension strain gauges 221. When the human body moves, the myoelectric signal electrodes 211 and The skin surface tension strain gauge 221 generates a weak current, which is filtered and amplified by a bandpass amplifier, and converted into a digital signal by the analog-to-digital conversion module 31. The MAP processes the current value and converts it into the corresponding joint angle to achieve whole-body posture tracking. Virtual touch is realized by a tights woven from hollow fiber tubes. Virtual touch is achieved by adjusting the pressure inside the local fiber tubes. The tights can achieve local pressure controllability. Each local pressure is controlled by a separate switch module, through the suction component. 44 introduces or exhausts gas to adjust local pressure, thereby realizing virtual touch, and virtual pain through the touch module 43 . At the same time, the IMU can record the attitude information of the main control box. The control unit 30 of the whole body posture tracking and tactile device 1 is connected to the VR device 2 through communication. The control unit 30 sends the user's body posture information to the VR device 2 for processing, and the VR device 2 sends the tactile point coordinates to the tactile control unit 40, thereby realizing Virtual haptic functionality. It should be noted that in this example, the first switch module 41 and the second switch module 42 are valves, the touch module 43 is a hammer, and the suction component 44 is an air pump.

In some examples, the whole-body posture tracking and haptic device 1 and the VR device 2 are connected through WIFI or Bluetooth to achieve information interaction.

In some examples, the VR device 2 includes a VR helmet. Of course, the VR device 2 can also be any device that is worn on the user's head and has a VR display, such as VR glasses.

It can be understood that the above embodiments are only exemplary embodiments adopted to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also regarded as the protection scope of the present invention.

Claims

A whole-body posture tracking and tactile device, which includes: a main structure, at least one action detection unit, a control unit, and at least one tactile control unit; wherein,

The main structure is configured for the user to wear on the body;

The action detection unit is installed on the main structure and configured to detect the user's body movements and generate user movement posture information;

The control unit is configured to process the received motion posture information, generate a first control signal, and transmit it to the VR device, so that the VR device can control the body movements of the virtual user, and after receiving the After the VR device feeds back the touch signal that the virtual user is touched, a second control signal is generated according to the touch signal;

The tactile control unit is configured to feedback real tactile sensation to the user's limb through the main structure according to the second control signal.
The whole body posture tracking and tactile device according to claim 1, wherein the movement posture information includes the rotation angle of the movement joint; the action detection unit includes a first detection module and a second detection module;

The first detection module is configured to detect the electromyographic signal generated by the user's skeletal muscles and process it;

The second detection module is configured to detect the user's skin surface tension signal and process it;

The control unit is configured to obtain the rotation angle of the user's moving joints based on the processed electromyographic signal and the skin surface tension signal and through a first preset algorithm, and obtain the rotation angle of the user's moving joints based on the processed Angle, through the second preset algorithm, the user's motion posture information is obtained to generate the first control signal.
The whole body posture tracking and tactile device according to claim 2, wherein the first detection module includes a myoelectric signal electrode and a first bandpass amplifier; the second detection module includes a skin surface tension strain gauge and a second band pass amplifier;

The electromyographic signal electrode is configured to detect the electromyographic signal generated by the user's skeletal muscles;

The first bandpass amplifier is configured to amplify the electromyographic signal and transmit it to the control unit;

The skin surface tension strain gauge is configured to detect the user's skin surface tension signal;

The second bandpass amplifier is configured to amplify the skin surface tension signal and transmit it to the control unit.
The whole-body posture tracking and haptic device according to claim 2 or 3, wherein the control unit includes an analog-to-digital conversion module, a first computing module and a first control module;

The analog-to-digital conversion module is configured to convert the processed electromyographic signal and the skin surface tension signal into a first digital signal and a second digital signal respectively;

The first calculation module is configured to obtain the rotation angle of the user's motion joint according to the first digital signal and the second digital signal through a first preset algorithm, and through a second preset algorithm, Obtain the user's movement posture information;

The first control module is configured to generate a first control signal according to the user's motion posture information and transmit it to the VR device so that the VR device can control the body movements of the virtual user, and after receiving After the VR device feeds back a touch signal indicating that the virtual user has been touched, a second control signal is generated based on the touch signal.
The whole-body posture tracking and haptic device according to claim 4, wherein the first calculation module is specifically configured to calculate based on the first digital signal and the second digital signal and through a first preset algorithm, Obtain the rotation angle of the user's moving joints, and calculate the spatial pose coordinates of the current position points of each moving joint relative to the initial point of each moving joint based on the pre-established kinematic model, thereby obtaining the user's moving posture. Information; wherein the initial point is the angle information of each moving joint of the user's initial posture.
The whole body posture tracking and tactile device according to claim 1, wherein the main structure is composed of a plurality of hollow wire tubes; the tactile control unit includes a first switch module, a second switch module and a touch module;

The first switch module and the second switch module are both connected to the hollow wire tube, and are used to control the entry and discharge of gas into the hollow wire tube according to the second control signal; and in the touch module It is provided on the hollow wire tube defined by the first switch module and the second switch module, and when the user wears the main structure, the touch module is located on the side of the hollow wire tube close to the human body. .
The whole body posture tracking and haptic device according to claim 6, wherein the control unit includes a second control module and a second computing module;

The second calculation module is configured to calculate the location information of the user actually being touched based on the touch signal of the virtual user being touched fed back by the VR device;

The second control module is configured to generate the second control signal according to the location information of the user's actual touch.
The whole-body posture tracking and haptic device according to claim 6, wherein the material of the hollow wire tube includes fiber material.
The whole-body posture tracking and haptic device of claim 6, wherein the first switch module and the second switch module include valves.
The whole-body posture tracking and haptic device according to claim 6, wherein the touch module includes a striking hammer.
The whole body posture tracking and haptic device according to claim 6, wherein the haptic control unit further includes a suction component configured to inflate gas into the hollow wire tube and remove the gas in the hollow wire tube. Gas is discharged.
The whole-body posture tracking and haptic device according to claim 11, wherein the suction component is an air pump.
A virtual reality system, which includes the whole body posture tracking and tactile device according to any one of claims 1 to 12 and a VR device; the VR device is communicatively connected with the whole body posture tracking and tactile device.
The virtual reality system according to claim 13, wherein the whole body posture tracking and tactile device and the VR device are connected through WIFI or Bluetooth.
The virtual reality system of claim 13, wherein the VR device includes a VR helmet.