WO2019019240A1

WO2019019240A1 - Device for integrating gesture recognition and ultrasonic tactile feedback and method and use thereof

Info

Publication number: WO2019019240A1
Application number: PCT/CN2017/098574
Authority: WO
Inventors: 袁昱
Original assignee: 深识全球创新科技（北京）有限公司
Priority date: 2017-07-25
Filing date: 2017-08-23
Publication date: 2019-01-31
Also published as: CN107340871A

Abstract

Provided in the invention is a device and method for integrating gesture recognition and ultrasonic tactile feedback. The device integrates functional units comprising a camera, an illumination unit and an ultrasonic nozzle assembly inside gloves, feeds back the hand operation in real time, achieves accurate recognition and tracing to the hand actions of a user and converts tactile information fed back into forces or force moments directly acting the hands of the user, so that an operator can produce a real touch feeling in a virtual environment, and meanwhile touch feeling at multiple angle and strengths can be also simulated, the gesture recognition speed and accuracy are improved, and the fidelity and immersion degree of virtual reality experience are increased. The invention also relates to use of the device for integrating gesture recognition and ultrasonic tactile feedback in virtual reality, augmented reality and mixed reality.

Description

Device for integrating gesture recognition and ultrasonic tactile feedback, method and use thereof

Technical field

The present invention relates to the field of virtual reality technologies, and in particular, to an apparatus for integrating gesture recognition and ultrasonic tactile feedback, and a method and use thereof.

Background technique

Virtual reality technology uses computer simulation technology to reproduce and restore real-world objects in a virtual world, and by setting certain interactive commands, users can interact with objects in the virtual world to immerse themselves in the virtual world. . Due to the highly immersive simulation of VR technology itself and its prominent role in defense, medicine, education, commerce, manufacturing and entertainment games, virtual reality technology is regarded as the most application-oriented and market-oriented technology in the 21st century. First, tactile feedback has always been considered an important part of achieving complete immersion in virtual reality.

The current lack of tactile simulation and single input in the virtual reality environment is considered to be an important issue affecting immersion. Lack of tactile feedback can make people feel that they are really holding virtual items. The existing mechanical haptic feedback scheme cannot achieve full coverage and can only provide a single force in the direction.

Some companies' virtual reality and augmented reality products use the controller of the controller for input, which is an expedient solution in the current situation where the gesture recognition scheme is still lacking. However, the VR handle also has obvious defects: the fine movements of the hand joints cannot be restored; the precise positioning of the hand movements cannot be performed; the accuracy is reduced by the influence of the surrounding environment.

At present, the skeleton-type pressure plate gloves can only place a pressure plate between each finger joint, which is completely different from the real real world. Only a small part of the simulated pressure can not guarantee the fidelity and the user experience is poor.

Gesture recognition is used by some companies in the field of virtual reality and augmented reality. However, simply recognizing the gesture by the camera requires placing the hand in the field of view of the camera and the user's hand needs to be lifted all the time, and cannot be used for a long time. Moreover, due to the complexity of the use environment, the surrounding environment is also very disturbed, which results in low recognition accuracy and slow recognition speed.

In order to ensure the fidelity and immersion of VR and AR interaction control, it is necessary to further improve the interactivity between users and the interactivity between the user and the identification device, and therefore, it is necessary to recognize the hand more accurately. Department of movement and movement details. The prior art recognizes the user's hand movement during the game by adding a three-dimensional camera module to the identification device, but often encounters problems such as low recognition accuracy and normal recognition at night.

Summary of the invention

It is an object of the present invention to at least solve one of the above disadvantages and disadvantages, which is achieved by the following technical solutions.

The present invention provides an apparatus for integrating gesture recognition and ultrasonic haptic feedback, comprising: a glove body and a plurality of functional units embedded in the glove body, the functional unit being used for a human hand in the glove body Gesture data acquisition and providing ultrasonic haptic feedback to the human hand based on the collected human hand gesture data.

Further, the plurality of functional units include a camera for acquiring human hand gesture data, a lighting unit that supplies light to the camera, and an ultrasonic showerhead assembly, wherein a first functional unit of the plurality of functional units includes the At least one of a camera, the illumination unit, and the ultrasonic showerhead assembly, a second one of the plurality of functional units including at least another of the camera, the illumination unit, and the ultrasonic showerhead assembly The third functional unit of the plurality of functional units includes at least one of the camera, the illumination unit, and the ultrasonic showerhead assembly.

Further, each functional unit includes a camera for acquiring human hand gesture data, a lighting unit that supplies light to the camera, and an ultrasonic showerhead assembly.

Further, the calculation module is further configured to perform calculation processing on the collected gesture data, perform gesture recognition, and generate gesture feedback information.

Further, the calculation module is further configured to perform calculation processing on the gesture data collected by the camera, perform gesture recognition, and generate gesture feedback information.

Further, the method further includes a control module and a communication module, wherein the control module is configured to receive the gesture feedback information and control the ultrasonic nozzle assembly to emit ultrasonic waves according to the gesture feedback information.

Further, the ultrasonic showerhead assembly includes at least one ultrasonic showerhead.

Further, the communication module is one or more of a USB module, a Bluetooth module, a GPRS/3G/4G/5G module, a WiFi module, a ZigBee module, or a network communication module of a specific protocol.

Further, the device for integrating gesture recognition and ultrasonic haptic feedback further includes a built-in battery And/or a power interface that can be connected to an external power source in a wired or wireless manner.

Further, the glove body is fixed at the wrist or the glove body is not fixed at the wrist.

Further, the glove body is a square body, a circular body or any other shape.

Further, the outer portion of the glove body is a flexible material or a rigid material.

Further, the inner surface of the glove body is covered with a monochromatic coating or a patterned coating.

Further, the terminal device is further included, and the terminal device is capable of receiving gesture data collected by the camera, performing gesture recognition, and generating gesture feedback information.

The present invention also provides a method for performing ultrasonic gesture recognition and haptic feedback using the above-described integrated gesture recognition and ultrasonic haptic feedback device, including:

Selecting a virtual environment and transmitting virtual environment information to a device that integrates gesture recognition and ultrasonic haptic feedback;

The device for integrating ultrasonic gesture recognition and haptic feedback collects a gesture of a human hand in the glove body in real time and transmits the acquired gesture information to the calculation module;

Using the computing module to perform gesture recognition and generate gesture feedback information, and transmitting ultrasonic waves according to the gesture feedback information to provide haptic feedback.

The present invention also provides another method for performing ultrasonic gesture recognition and haptic feedback using the above-described integrated gesture recognition and ultrasonic haptic feedback device, including:

The device for integrating gesture recognition and ultrasonic haptic feedback collects a gesture of a human hand in the glove body in real time and transmits the acquired gesture information to the terminal device;

Using the terminal device to perform gesture recognition and generate gesture feedback information, and transmit the gesture feedback information to the integrated ultrasonic gesture recognition and haptic feedback device;

The device for recognizing the ultrasonic haptic feedback with the integrated gesture receives the gesture feedback information, and transmits the ultrasonic wave according to the gesture feedback information to provide haptic feedback.

Further, the method further includes adjusting a transmission mode of the ultrasonic wave according to the gesture feedback information.

Further, adjusting the transmission mode of the ultrasonic wave includes adjusting one or more of a pulse of the ultrasonic wave, adjusting a frequency of the ultrasonic wave, adjusting an emission direction of the ultrasonic wave, and adjusting an intensity of the ultrasonic wave.

The present invention also relates to the above apparatus for integrating gesture recognition and ultrasonic haptic feedback in virtual reality, Use in augmented reality or mixed reality.

The advantages of the invention are as follows:

(1) The tactile feedback of the present invention has a strong sense of reality, and can simulate the touch of various angles and strengths, further increasing the fidelity and immersion of the virtual reality experience.

(2) The integrated gesture recognition of the present invention can make the use environment simple and simplify the algorithm, thereby improving the recognition speed and the recognition accuracy.

(3) The present invention integrates gesture recognition and tactile feedback into the glove, and the device has high integration, which can perfectly replace the handle controller.

(4) The present invention is small in size, light in weight, and can be worn and operated for a long time.

DRAWINGS

Various other advantages and benefits will become apparent to those skilled in the art from a The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting. Throughout the drawings, the same reference numerals are used to refer to the same parts.

FIG. 1 is a schematic diagram of assembly of an apparatus for integrating gesture recognition and ultrasonic haptic feedback according to an embodiment of the present invention.

2 is a schematic structural diagram of functional units of an apparatus for integrating gesture recognition and ultrasonic haptic feedback according to an embodiment of the present invention.

FIG. 3 is another schematic structural diagram of a functional unit of an apparatus for integrating gesture recognition and ultrasonic haptic feedback according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of another functional unit of an apparatus for integrating gesture recognition and ultrasonic haptic feedback according to an embodiment of the present invention.

FIG. 5 is a schematic structural diagram of an apparatus for integrating gesture recognition and ultrasonic haptic feedback according to an embodiment of the present invention.

FIG. 6 is a flowchart of operations of an apparatus for integrating gesture recognition and ultrasonic haptic feedback according to an embodiment of the present invention.

FIG. 7 is another flowchart of an operation of an apparatus for integrating gesture recognition and ultrasonic haptic feedback according to an embodiment of the present invention.

The reference numerals in the figure are as follows:

1-Glove body 2-functional unit

201-ultrasonic probe 202-camera

203-Lighting unit 3-Control module

4-communication module 5-calculation module

6-IC board

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While the exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the invention may be embodied in various forms and not limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be more fully understood, and the scope of the disclosure can be fully conveyed to those skilled in the art.

FIG. 1 is a schematic diagram of an apparatus for integrating gesture recognition and ultrasonic tactile feedback according to an embodiment of the present invention. As shown in FIG. 1 , an apparatus for integrating gesture recognition and ultrasonic tactile feedback based on ultrasonic waves according to the present invention includes a glove. The body 1 and a plurality of functional units 2 embedded in the glove body 1 are used for performing gesture data collection on a human hand in the glove body 1 and providing ultrasonic tactile feedback to the human hand based on the collected human hand gesture data.

2 is a schematic structural diagram of a functional unit of an apparatus for integrating gesture recognition and ultrasonic haptic feedback based on ultrasonic waves according to an embodiment of the present invention. As shown in FIG. 2, each functional unit 2 includes a camera for acquiring human hand gesture data. 201. A lighting unit 202 and an ultrasonic head assembly 203 that provide light to the camera. The ultrasonic head assembly 203 performs ultrasonic wave emission at different angles and intensities according to the position of the hand and the finger provided by the camera 201 to form a tactile sensation.

The glove body 1 is internally covered with the functional unit 2, and the user can use the functional unit 2 to project the touch of pressing the virtual object onto the user's hands, so that the user can obtain tactile feedback in midair without touching the screen.

In this embodiment, the functional units 2 are evenly distributed in various parts inside the glove body 1. In another embodiment, the functional units 2 of certain parts may be appropriately added and deleted. For example, the finger and the palm portion are more active, and the back of the hand is less active, and the arrangement of the functional unit 2 at the back of the hand can be appropriately reduced. The addition and deletion combination of the functional unit 2 includes but is not limited to the embodiment.

In addition, as shown in FIG. 4, the functional units 2 are not identical, some of which only include the camera 201, others only include the illumination unit 202, and some only include the ultrasonic nozzles 203, and their number and position distribution are not exactly the same. , can be set as needed.

The first functional unit of the plurality of functional units 2 includes at least one of a camera 201, a lighting unit 202, and an ultrasonic showerhead assembly 203. The second functional unit of the plurality of functional units 2 includes a camera 201, a lighting unit 202, and an ultrasonic nozzle. At least another of the components 202, the third of the plurality of functional units 2 includes at least one of the camera 201, the illumination unit 202, and the ultrasonic showerhead assembly 203.

The device for ultrasonic haptic feedback composed of the first functional unit, the second functional unit, and the third functional unit is capable of performing complete gesture recognition and haptic feedback.

In another embodiment, the plurality of functional units 2 may also be mounted on a single integrated circuit board 6, and the integrated circuit board 6 may be disposed inside the glove body 1. As shown in FIG. 4, a plurality of functional units 2 are arranged on the integrated circuit board 6, and the functional units 2 are collectively disposed on a single integrated circuit board 6, so that electrical connection can be reduced.

According to an embodiment of the present disclosure, the illumination unit 202 of the functional unit 2 may emit visible light or invisible light to track the recognition gesture action. For example, the illumination unit 202 may emit infrared light, and the camera 201 may accurately track the recognition gesture even during the daytime. Information, unaffected by light. In addition, the illumination unit 202 can emit light of different colors, frequencies or brightness according to different environments to be tracked and recognized by the camera 201. For example, during the daytime, the illumination unit 202 emits white light due to sufficient light, and the camera 201 cannot accurately track the recognition gesture information. If the red light is emitted, the camera 201 can accurately track the recognition gesture information; at night, due to the dim light, the illumination unit 202 can emit white light to be tracked and recognized by the camera 201; for example, in the game, according to the game progress content, In a fierce battle, the lighting unit 202 emits a pulsed flashing red light. In the peace phase, the lighting unit 202 emits a pulsed blue light; in the indoor game, the lighting unit 202 emits a low-intensity light, and when the game scene is transferred to the outdoor, the lighting is illuminated. Unit 202 emits light of a higher brightness and the like.

Each of the ultrasonic nozzles of the ultrasonic showerhead assembly 203 can eject a controlled force so that the ultrasonic waves can cover all of the limbs it surrounds. At the same time, each nozzle is relatively small, can be perfectly distributed to the surface of various shapes, so that the hand feels the feedback of the force, bringing a more realistic experience to the user, improving the fidelity of the virtual reality experience, for example, in the game. , the two players fight in the virtual world, once When the other party is hit, or if it is hit by itself, the ultrasonic nozzle will stimulate the skin to feel tactile. When you are hit, the hand will feel tactile and feel like it was actually hit.

The number of ultrasonic head assemblies 203 on each functional unit 2 can be set as needed, as shown in FIG. 3, which shows a schematic structural view of another embodiment of the functional unit 2 of the present invention. The ultrasonic nozzle assembly 203 is composed of 3 Each of the ultrasonic nozzles can eject a controlled force to stimulate the neurons in the hand, creating different touches in the hand.

The user can also adjust the effects of the haptic feedback, such as adjusting the intensity of the ultrasonic pulse or frequency-weakening feedback, or increasing the duration of the haptic feedback to meet the personalized user experience needs.

The camera 201 and the ultrasonic head assembly 203 are integrated into the functional unit 2, which can well track each joint of the human hand, and accurately capture various gestures and movements of the human hand, and can recognize and track the hand movement all day and night, regardless of day and night. Implement gesture recognition and control. It also enables users to interact with virtual objects in the virtual environment to produce more natural and immersive tactile feedback, greatly improving the user experience. The present invention combines gesture recognition tracking and tactile feedback in a compact and comfortable device that makes the virtual reality experience more interactive, more immersive and more immersive.

FIG. 5 is a schematic structural diagram of an integrated ultrasonic touch and gesture recognition apparatus according to an embodiment of the present invention. As shown in FIG. 5 , the glove body 1 further includes a control module 3 , a communication module 4 , and a calculation module 5 .

The calculation module 5 is electrically connected to the control module 3, and is configured to perform calculation and processing on the spatial position information and the gesture information of each part of the collected human hand through an algorithm, accurately locate and recognize the human hand gesture, and generate gesture feedback information. The calculation process of the recognition of the gesture by the calculation module 5 is not limited to one or several specific types of algorithms. The calculation module 5 can easily identify the position of the user's finger, improve the accuracy of the tracking recognition, conveniently model the user's finger in the virtual environment, improve the fidelity of the virtual environment, and enhance the user experience.

The control module 3 is mounted on the back of the glove body 1. The control module 3 is an integrated circuit board and a main control chip, and each functional unit 2 is electrically connected through a superfine wire. The control module 3 is configured to receive the human hand gesture recognition feedback information provided by the calculation module 5 and control the ultrasonic nozzle assembly 203 to emit ultrasonic waves according to the gesture feedback information to generate haptic feedback.

In another embodiment, when the device performs gesture recognition, the human hand gesture data collected by the camera 201 can be directly transmitted to the terminal device through the communication module 4 without using the calculation module 5, and the gesture preset by the terminal device is used for gesture recognition. .

The human hand gesture data collected by the camera 201 is transmitted to the terminal device through the communication module 4, and the terminal device combines the collected human hand gesture data with the virtual environment information, and generates a human hand gesture feedback information by using a program algorithm built in the terminal device, and the human hand is generated. The gesture feedback information is sent to the control module 3 through the communication module 4, and the control module 3 sends a tactile command to the function unit 2 according to the received human hand gesture feedback information, and the ultrasonic nozzle assembly 203 of the function unit 2 emits ultrasonic waves from the respective directions to the hand. In this way, a tactile sensation is formed, and the tactile sensation generated in the virtual environment is fed back to the operator, giving the operator an immersive immersive feeling.

The communication module 4 is fixed to the control module 3, and is used for connecting the device and the terminal device for integrating gesture recognition and ultrasonic haptic feedback to realize communication and data exchange. The communication module 4 is a USB module, a Bluetooth module, and a GPRS/3G/4G/5G module. One or more of a WiFi module, a ZigBee module, or a network communication module of a specific protocol. In the preferred embodiment, setting the communication module 4 to the wireless mode allows the device to be free from space constraints and increases the degree of freedom of use.

The power supply of the control module 3 and the functional unit 2 can be supplied by the built-in battery in the glove, or can be connected to the external power supply through a wired or wireless power interface. The invention is not limited thereto.

The glove body 1 is used for mounting and fixing the functional unit 2 and other modules, and can be worn on the wrist of the user. The glove body 1 can be fixed to the human hand at the wrist, and the human hand moves inside the glove body 1. The design of fixing the glove body 1 to the wrist can effectively prevent the glove from being too loose and lose the immersion feeling when the user experiences, and the curvature of the finger joint and the strength of the finger can be accurately conveyed, so that the user naturally interacts with the virtual object.

In other embodiments of the present invention, the glove body 1 may not be fixed to the wrist, and is directly disposed in a fixed state, that is, the glove body 1 is fixedly disposed on the table top, and the human hand moves in the glove body 1 at this time. The user in front of the computer grabs the object, making the user's finger more flexible, and the VR experience process is more natural and comfortable.

The glove body 1 may be a square body, a circular body or any other shape, and the present invention is not limited thereto as long as accurate recognition tracking and tactile feedback of the hand movement can be achieved. For example, it may be a flexible glove type, or it may be a spherical shape, or may not even be fully enclosed (for example, a semi-enclosed hand guard of a Western foil).

The outer part of the glove body 1 is made of a flexible material or a rigid material, and the glove body 1 uses a breathable flexible material, which allows the user to wear more comfortable and facilitates flexible movement of the hand. When the glove body 1 is set to a fixed state, a rigid material can be conveniently fixed.

The inner surface of the glove body 1 is covered with a monochromatic coating or a patterned coating, which is provided as a patterned coating. These patterns and the markings contained therein will facilitate the relative and absolute positioning of each part of the human hand during gesture recognition. position.

As shown in FIG. 6, according to an embodiment of the present disclosure, a method for performing tactile feedback and gesture recognition using the integrated gesture recognition device with ultrasonic haptic feedback includes the following steps:

1) selecting a virtual environment and transmitting the virtual environment information to a device that integrates gesture recognition and ultrasonic haptic feedback;

Specifically, after the power is turned on, the device integrating the gesture recognition and the ultrasonic haptic feedback starts to work, and the appropriate virtual scene information is selected on the software corresponding to the device.

2) The device for integrating gesture recognition and ultrasonic haptic feedback collects the gesture of the human hand in the glove body in real time and transmits the acquired gesture information to the calculation module;

The human hand moves in the device integrating the gesture recognition and the ultrasonic haptic feedback, and the camera 201 of the function unit 2 collects the human hand gesture information in real time, and transmits the acquired human hand gesture information to the calculation module 5 to provide accurate gesture recognition tracking.

3) Performing gesture recognition using the calculation module and generating gesture feedback information, and transmitting the ultrasonic wave according to the gesture feedback information to provide tactile feedback.

Specifically, after receiving the gesture motion data collected by the camera 201, the calculation module 5 performs unified analysis processing by using an algorithm, performs gesture recognition, and generates gesture feedback information.

After receiving the gesture feedback information, the control module 3 of the device for integrating gesture recognition and ultrasonic haptic feedback controls the ultrasonic head assembly 203 of the functional unit 2 to emit ultrasonic waves from different angles to the hand, thereby forming a tactile sensation, and The tactile sensation generated in the virtual environment is fed back to the operator, giving the operator an immersive immersion.

Adjusting the emission mode of the ultrasonic wave can adjust the effect of the tactile feedback to meet the personalized user experience. The method of adjusting the emission of the ultrasonic wave includes adjusting one or more of the pulse of the ultrasonic wave, adjusting the frequency of the ultrasonic wave, adjusting the emission direction of the ultrasonic wave, and adjusting the intensity of the ultrasonic wave.

As shown in FIG. 7 , in another embodiment, the device for integrating gesture recognition and ultrasonic haptic feedback is only responsible for the collection of gesture data, and the process of gesture recognition occurs at the terminal device, and the method for haptic feedback and gesture recognition includes the following steps. :

1) Select the virtual environment and send the virtual environment information to the integrated gesture recognition and ultrasonic tactile counter Feeding device.

2) The device integrating the gesture recognition and the ultrasonic haptic feedback collects the gesture data of the human hand in the glove body in real time and transmits the acquired gesture information to the terminal device.

After receiving the gesture motion data collected by the camera 201, the terminal device performs unified analysis processing.

3) Using the terminal device for gesture recognition and generating gesture feedback information, and transmitting the gesture feedback information to the device for integrating gesture recognition and ultrasonic tactile feedback.

Specifically, the terminal device combines the human hand gesture information and the virtual environment information received by the device that integrates the gesture recognition with the ultrasonic haptic feedback to perform data processing and analysis, performs gesture recognition, and generates gesture feedback information. At the same time, when the terminal device detects that the hand needs to have tactile feedback (for example, in a virtual scene, when the hand touches the virtual object, the object will give the hand a force tactile feedback), the terminal device will give the gesture feedback. Information is sent via communication module 4 to a device that integrates gesture recognition and ultrasonic haptic feedback.

4) The device that uses the integrated gesture recognition and the ultrasonic haptic feedback receives the gesture feedback information, and emits the ultrasonic wave according to the gesture feedback information to provide the haptic feedback.

After receiving the gesture feedback information, the control module 3 of the device for integrating gesture recognition and ultrasonic haptic feedback controls the ultrasonic nozzle assembly 203 of the functional unit 2 to emit ultrasonic waves from various orientations to the hand, thereby forming a tactile sensation, and The tactile sensation generated in the virtual environment is fed back to the operator, giving the operator an immersive immersive feel.

For ease of understanding, the following describes the embodiment in combination with a specific application scenario:

First, the user connects the device that integrates gesture recognition and ultrasonic haptic feedback to the computer via WiFi, and selects appropriate scene information on the computer corresponding software, such as the earth model appearing on the screen. In order to find the position of a certain place on the earth model, the user performs an action of “pressing the earth”, and the function unit 2 in the device for integrating gesture recognition and ultrasonic tactile feedback worn by the user tracks the hand motion and acquires the hand motion information. And sent to the computer in real time, the computer combines the hand motion information with the virtual scene information to obtain the gesture feedback information, and then the computer sends the gesture feedback information to the device integrating the gesture recognition and the ultrasonic haptic feedback, so that the integrated gesture recognition and the ultrasonic touch are integrated. The ultrasonic nozzle assembly 203 in the feedback device emits ultrasonic waves according to the gesture hand feedback information, so that the user feels strong pressure on the hand, resulting in a tactile sensation similar to the touch earth model. At the same time, users can also make The integrated gesture recognition and ultrasonic haptic feedback device moves and lifts the earth model in the virtual scene, and generates a real feeling of moving or lifting.

In this embodiment, the haptic feedback is added in the visual virtual scene, and the interaction with the virtual scene is realized through the activity of the hand, thereby providing the operator with a more direct and more natural human-computer interaction mode, thereby improving the user experience.

In another embodiment of the invention, a device that integrates gesture recognition and ultrasonic haptic feedback can be utilized to assist the patient in healing. Traditional rehabilitation programs use 2D images to help patients recover from exercise after surgery. For example, an apple is displayed on a 2D screen, allowing the rehabilitated patient to grab the hand and display a reward after the capture. The invention can make the patient have a real sense of distance to catch the apple and have tactile feedback.

The device for integrating gesture recognition and ultrasonic haptic feedback provided by the invention can also be used together with other virtual reality devices, and the virtual scene information is projected and displayed by the glasses, for example, when the user wears the virtual reality glasses and operates the computer, the desk is obtained. The virtual scene information of the study room, the virtual environment will be displayed in real time in the virtual reality glasses. When the user touches the desk, the 3D scene picture presentation can be seen not only through the virtual reality glasses, but also the device based on the integrated gesture recognition and ultrasonic tactile feedback based on the ultrasonic wave. Feel the action of touching the desk, allowing users to better feel the changes in the virtual environment and increase the interaction with the virtual environment.

The device for integrating gesture recognition and ultrasonic haptic feedback provided by the invention integrates gesture recognition and haptic feedback into a micro function unit, and embeds the function unit inside the glove body, and feedbacks the hand operation in real time to realize the user's hand. Accurate recognition and tracking of motion, and convert the tactile information feedback from the terminal device into force or moment directly acting on the human hand, so that the operator produces an "immersive" tactile presence effect, thereby realizing the reality in the virtual environment. The touch feel, and it overcomes the shortcomings of the mechanical structure of the handle controller, the small range of hand movement and the lack of force feedback function of the traditional data glove, and has wide application prospects, which can be applied to virtual reality, augmented reality and mixed reality. Among various technologies.

According to an embodiment of the present disclosure, the terminal device may be a device having a communication function. For example, the terminal device can be a desktop PC computer, a smart phone, a tablet computer, a portable multimedia player, a portable medical device, a digital camera, or a wearable device.

It should be noted that the actions indicated in the exemplary method may not be as shown in the drawings or in the specification. The sequence occurs. For example, two consecutive actions may be executed substantially concurrently at the time of execution, or some of the actions may be performed in the reverse order, depending on the functionality involved.

It can be understood that in order to implement the above gesture recognition and haptic feedback, a corresponding set of drivers and application software (not shown) must be completed.

The above is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or within the technical scope disclosed by the present invention. Alternatives are intended to be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the appended claims.

Claims

An apparatus for integrating gesture recognition and ultrasonic haptic feedback, comprising:

Glove body;

And a plurality of functional units embedded on the glove body, the functional unit is configured to perform gesture data collection on a human hand in the glove body and provide ultrasonic tactile feedback to the human hand according to the collected human hand gesture data.
The apparatus for integrating gesture recognition and ultrasonic haptic feedback according to claim 1, wherein the plurality of functional units comprise a camera for acquiring human hand gesture data, a lighting unit for providing light to the camera, and an ultrasonic nozzle assembly. The first functional unit of the plurality of functional units includes at least one of the camera, the illumination unit, and the ultrasonic showerhead assembly, and the second functional unit of the plurality of functional units includes At least another one of the camera, the illumination unit, and the ultrasonic showerhead assembly, the third of the plurality of functional units including at least the camera, the illumination unit, and the ultrasonic showerhead assembly The other one.
The apparatus for integrating gesture recognition and ultrasonic haptic feedback according to claim 1, wherein each functional unit comprises a camera for acquiring human hand gesture data, a lighting unit for providing light to the camera, and an ultrasonic head assembly.
The device for integrating gesture recognition and ultrasonic haptic feedback according to claim 1, further comprising a calculation module, wherein the calculation module is configured to perform calculation processing on the collected gesture data, perform gesture recognition, and generate gesture feedback. information.
The device for integrating gesture recognition and ultrasonic haptic feedback according to claim 2, further comprising a calculation module, wherein the calculation module is configured to perform calculation processing on the gesture data collected by the camera, and perform gesture recognition, and Generate gesture feedback information.
The device for integrating gesture recognition and ultrasonic haptic feedback according to claim 5, further comprising a control module and a communication module, wherein the control module is configured to receive the gesture feedback information and control the location according to the gesture feedback information The ultrasonic head assembly emits ultrasonic waves.
The apparatus for integrating gesture recognition and ultrasonic haptic feedback according to claim 2, wherein said ultrasonic head assembly comprises at least one ultrasonic head.
The device for integrating gesture recognition and ultrasonic haptic feedback according to claim 6, wherein the communication module is a USB module, a Bluetooth module, a GPRS/3G/4G/5G module, a WiFi module, a ZigBee module or a specific protocol. One or more of the network communication modules.
Apparatus for integrating gesture recognition and ultrasonic haptic feedback according to any one of claims 2 to 8, wherein said means for integrating gesture recognition and ultrasonic haptic feedback further comprises a built-in battery and/or is capable of externally A power connector that is connected in a wired or wireless manner.
The apparatus for integrated gesture recognition and ultrasonic haptic feedback according to any one of claims 1 to 8, wherein the glove body is fixed at the wrist or the glove body is not fixed at the wrist.
The device for integrating gesture recognition and ultrasonic haptic feedback according to any one of claims 1 to 8, wherein the glove body is a square body, a circular body or any other shape.
The device for integrating gesture recognition and ultrasonic haptic feedback according to any one of claims 1 to 8, wherein the outer portion of the glove body is a flexible material or a rigid material.
The apparatus for integrated gesture recognition and ultrasonic haptic feedback according to any one of claims 1 to 7, wherein the inner surface of the glove body is covered with a monochromatic coating or a patterned coating.
The device for integrating gesture recognition and ultrasonic haptic feedback according to claim 4 or 5, further comprising a terminal device, wherein the terminal device is capable of receiving gesture data collected by the camera, performing gesture recognition, and generating Gesture feedback information.
A method for performing ultrasonic gesture recognition and haptic feedback using the integrated gesture recognition and ultrasonic haptic feedback device according to any one of claims 1 to 14, characterized in that it comprises:

Selecting a virtual environment and transmitting virtual environment information to a device that integrates gesture recognition and ultrasonic haptic feedback;

The device for integrating ultrasonic gesture recognition and haptic feedback collects a gesture of a human hand in the glove body in real time and transmits the acquired gesture information to the calculation module;

Using the computing module to perform gesture recognition and generate gesture feedback information, and transmitting ultrasonic waves according to the gesture feedback information to provide haptic feedback.
A method for performing ultrasonic gesture recognition and haptic feedback using the integrated gesture recognition and ultrasonic haptic feedback device according to any one of claims 1 to 14, characterized in that it comprises:

Selecting a virtual environment and transmitting virtual environment information to a device that integrates gesture recognition and ultrasonic haptic feedback;

The device for integrating gesture recognition and ultrasonic haptic feedback collects a gesture of a human hand in the glove body in real time and transmits the acquired gesture information to the terminal device;

Using the terminal device to perform gesture recognition and generate gesture feedback information, and transmit the gesture feedback information to the integrated ultrasonic gesture recognition and haptic feedback device;

The device for recognizing the ultrasonic haptic feedback with the integrated gesture receives the gesture feedback information, and transmits the ultrasonic wave according to the gesture feedback information to provide haptic feedback.
The method of ultrasonic gesture recognition and haptic feedback according to claim 15 or 16, further comprising adjusting a transmission mode of the ultrasonic wave according to the gesture feedback information.
The ultrasonic gesture recognition and tactile feedback method according to claim 17, wherein the adjusting the ultrasonic wave transmission mode comprises: adjusting a pulse of the ultrasonic wave, adjusting a frequency of the ultrasonic wave, adjusting an emission direction of the ultrasonic wave, and adjusting an intensity of the ultrasonic wave. Or a variety.
The use of an apparatus for integrated gesture recognition and ultrasonic haptic feedback according to any one of claims 1 to 14 in virtual reality, augmented reality or mixed reality.