WO2018098861A1

WO2018098861A1 - Gesture recognition method and device for virtual reality apparatus, and virtual reality apparatus

Info

Publication number: WO2018098861A1
Application number: PCT/CN2016/111062
Authority: WO
Inventors: 张茜; 张绍谦; 张超
Original assignee: 歌尔科技有限公司
Priority date: 2016-11-29
Filing date: 2016-12-20
Publication date: 2018-06-07
Also published as: CN106845335A; CN106845335B

Abstract

A gesture recognition method and device for a virtual reality apparatus, and a virtual reality apparatus. The gesture recognition method comprises: controlling a depth camera to capture a current hand image of a current user (S110); determining, according to the current hand image, whether the current user executes a tapping action or not (S120); if yes, extracting current features from the current hand image (S130); and matching the current features with reference features in a model, and determining, according to the matching result, a key tapped by the current user (S140). By applying a virtual keyboard to reality, the gesture recognition method improves the use flexibility of users, and also releases space of a real keyboard, and reduces the trouble of the user caused by complex character input, thereby improving user experience.

Description

Gesture recognition method, device and virtual reality device for virtual reality device

Technical field

The present invention relates to the field of virtual reality device technologies, and in particular, to a gesture recognition method, apparatus, and virtual reality device for a virtual reality device.

Background technique

Virtual Reality (VR) is a high-tech that has emerged in recent years. Virtual reality technology will be a key technology to support a comprehensive integration of multidimensional information space combining qualitative and quantitative, perceptual knowledge and rational understanding. As the speed of the Internet increases, an Internet era based on virtual reality technology is quietly coming, which will dramatically change people's production and lifestyle. It is conceivable that we can experience and interact with the virtual world through VR headsets, such as swimming in space, skydiving, and so on.

At present, the emerging virtual reality technology has penetrated into the fields of office and entertainment, which has caused changes in many industries.

Summary of the invention

According to a first aspect of the present invention, a gesture recognition method for a virtual reality device is provided, the virtual reality device including a depth camera, and the gesture recognition method includes:

Controlling the depth camera to capture a current hand image of the current user;

Determining, according to the current hand image, whether the current user performs a tap action, and if so, then:

Extracting a current feature from the current hand image;

Matching the current feature with a reference feature in the model, and determining a button of the current user tap based on the matching result.

According to a second aspect of the present invention, a gesture recognition apparatus for a virtual reality device is provided, including:

a first control module, configured to control the depth camera to collect a current hand image of the current user;

a determining module, configured to determine, according to the current hand image, whether the current user performs a tapping action;

a current feature extraction module, configured to extract a current feature from the current hand image if the determination result of the determination module is YES;

And a matching module, configured to match the current feature with a reference feature in the model, and determine, according to the matching result, the button that the current user taps.

According to a third aspect of the invention, there is provided a virtual reality device comprising the gesture recognition device according to the second aspect of the invention.

According to a fourth aspect of the present invention, there is provided a virtual reality device comprising a depth camera for acquiring an image, the memory for storing an instruction, the instruction for controlling the The processor performs the gesture recognition method according to the first aspect of the present invention.

According to a fifth aspect of the invention, there is provided a computer readable storage medium storing program code for performing the gesture recognition method according to the first aspect of the invention.

Other features and advantages of the present invention will become apparent from the Detailed Description of the <RTIgt;

DRAWINGS

The accompanying drawings, which are incorporated in FIG

1 is a flow chart of an embodiment of a gesture recognition method for a virtual reality device in accordance with the present invention;

2 is a block schematic diagram of an implementation structure of a gesture recognition apparatus for a virtual reality device according to the present invention;

3 is a block schematic diagram of an implementation structure of a virtual reality device in accordance with the present invention.

detailed description

Various exemplary embodiments of the present invention will now be described in detail with reference to the drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in the embodiments are not intended to limit the scope of the invention unless otherwise specified.

The following description of the at least one exemplary embodiment is merely illustrative and is in no way

Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

In all of the examples shown and discussed herein, any specific values are to be construed as illustrative only and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

In order to solve the problem that the interaction of the head-mounted virtual reality device existing in the prior art is mainly through language, gesture, etc., the complex processing of the text cannot be realized, and a gesture recognition method for the virtual reality device is provided. Reality devices include depth cameras.

The depth camera is also called a depth sensor or a 3D sensor. For example, it can be a TOF camera that emits modulated near-infrared light. When it is reflected by an object, the camera calculates the distance of the object by calculating the time difference or phase difference between the light emission and reflection. In order to generate depth information, in addition to the traditional camera shooting, the three-dimensional contour of the object can be represented by images of different colors representing different distances.

1 is a flow chart of an embodiment of a gesture recognition method for a virtual reality device in accordance with the present invention.

According to FIG. 1, the gesture recognition method comprises the following steps:

Step S110, controlling the depth camera to collect the current hand image of the current user.

Specifically, for example, the TOF camera may be controlled to emit modulated near-infrared light, and the camera may reflect the distance between the current user's hand and the phase difference by calculating the light emission and reflection time difference or phase difference. In order to generate depth information, and in combination with the traditional camera shooting, the three-dimensional contour of the hand can be represented by different colors to represent the current hand of different distances. The image is presented.

Step S120: determining, according to the current hand image, whether the current user performs a tap action, and if yes, executing step S130; if not, proceeding to step S110.

According to the depth camera, the curvature of each point of the hand contour in the current hand image is obtained, and the fingertip position can be calculated. For each point of the hand contour in the current hand image, the curvature of each point is determined according to a certain step size. The curvature of the fingertip has a certain range, and it is determined whether the curvature of each point is within the range by checking the curvature of each point. The position of the fingertip can be determined, and the position of the other key points of the hand can be estimated by the morphology by the result of the gesture determination and the position of the fingertip, wherein the other key points can be, for example, a joint.

The depth sensor can be placed on top of the virtual reality device or anywhere in front of the user's hand. In a specific embodiment of the invention, the location of the depth sensor may be located on a virtual reality device. After the depth sensor collects the current hand image of the current user, the current fingertip coordinates of the current user's hands can be obtained by calculating the curvature of the hand contour point in the current hand image, and other key points of the user hand are calculated according to the user's fingertip coordinates. Point, hand fingertips and other key points are imaged by the user's hand image on top of the VR keyboard image by, for example, 3D rendering techniques. Then, based on the current hand image, the depth change of the user's finger can be confirmed to confirm whether or not the keyboard is tapped.

Using 3D rendering technology, the image is clear, the resolution of the image is increased, and the viewing angle and visual distance can be adjusted appropriately.

Since the depth sensor is obtained as a depth map, the depth is different according to the distance of the hand from the depth sensor. Therefore, when the finger is raised, the depth value is small, and when the finger is dropped, the depth value is large, thereby determining whether the current user performs. Tap the action.

In a specific embodiment of the present invention, the virtual reality device further includes a display screen, and before performing step S120, the method further includes:

The keyboard image and the initial position of the current user's finger on the keyboard image are displayed on the display.

When the current user uses the virtual device, for example, a head-mounted virtual reality device for character input, in order to alleviate the discomfort of the current user's arm, the current user can use the surrounding flat object such as a desk or the like. The current user's hands can be placed according to the gesture of the physical keyboard, depth camera The head obtains the current hand image and fingertip coordinates, which can be placed by the index finger fingertip normalization processing to place the left index finger on the keyboard button F, and the right hand index finger is placed on the keyboard button J, the current user can be based on the virtual reality Imaging of the device, appropriate adjustment of other finger positions, to fall to the correct initial position of each finger.

In this way, the current user can clearly see the position of the button on the keyboard that needs to be pressed, thereby determining the action of the tap and improving the user experience.

Step S130, extracting a current feature from the current hand image.

In a specific embodiment of the invention, the extraction of current features may be implemented by a neural network (CNN) improved algorithm. The CNN improved algorithm obtains all the neurons of the hand according to the convolution and the feature points of the user's hand (for example, 10), and then according to the maximum pool layer and the fully connected layer, wherein the CNN improvement algorithm can be adopted, for example. Caffe or the algorithm architecture of CNN such as tensorflow, and extract the current features from it.

In step S140, the current feature is matched with the reference feature in the model, and the button of the current user tap is determined according to the matching result.

The reference feature may be, for example, stored in the model before the virtual reality device is shipped from the factory, or may be stored by the current user before using the virtual reality device.

In a specific embodiment of the present invention, before performing step S140, the gesture recognition method further includes:

Controlling the depth camera to acquire a reference hand image of the reference user;

Extracting the reference feature from the reference hand image and storing the reference feature in the model.

Specifically, the reference hand image of the user's various actions of tapping the keys of the keyboard is collected, and the reference features are extracted from each reference hand image to establish a reference feature corresponding to various actions of the hand. The model, the resulting model is applied to match the current feature, and if the match is successful, the button of the current user tap corresponding to the matching reference feature can be determined.

After performing step S140, the gesture recognition method further includes:

According to the button that the current user taps, the current position of the current user's finger on the keyboard image is displayed on the display screen.

Specifically, the method of displaying the current position of the current user's finger on the keyboard image on the display screen may be the same as the above method of displaying the initial position.

In this way, the present invention can distinguish the various gestures, the left and right hand confirmation, and the fingertip coordinate acquisition by using the gesture recognition technology of the depth sensor, and the virtual keyboard can be created by the finger tapping the keyboard screen in front of the user to complete the character or the digital input. In the real world, it can greatly improve the flexibility of user use, and also release the physical keyboard space, reducing the user's trouble caused by complex character input, thereby improving the user experience.

The present invention also provides a gesture recognition apparatus for a virtual reality device, and FIG. 2 is a block schematic diagram of an implementation structure of a gesture recognition apparatus for a virtual reality device according to the present invention.

According to FIG. 2 , the gesture recognition apparatus 200 includes a first control module 210 , a determination module 220 , a current feature extraction module 230 , and a matching module 240 .

The first control module 210 is configured to control the depth camera to collect the current hand image of the current user.

The determining module 220 is configured to determine, according to the current hand image, whether the current user performs a tapping action.

The current feature extraction module 230 is configured to extract the current feature from the current hand image if the determination result of the determination module is YES.

The matching module 240 is configured to match the current feature with the reference feature in the model, and determine a button of the current user tap according to the matching result.

Specifically, the gesture recognition apparatus further includes a second control module and a reference feature extraction module, wherein the second control module is configured to control the depth camera to acquire the reference hand image of the reference user; and the reference feature extraction module is configured to extract the reference hand image. Refer to the feature and store the reference feature in the model.

Further, the virtual reality device further includes a display screen, and the gesture recognition device further includes a first display module configured to display the keyboard image and the initial position of the current user's finger on the keyboard image on the display screen.

On the basis of the above, the gesture recognition device further includes a second display module, configured to display, on the display screen, the current user's finger on the keyboard according to the button pressed by the current user determined by the matching module. Like the current location on.

The present invention also provides a virtual reality device. According to one aspect, the virtual reality device includes a gesture recognition apparatus 200 for a virtual reality device of the present invention. The virtual reality device may be, for example, a product such as virtual reality glasses, a virtual reality helmet, or the like.

3 is a block schematic diagram of an implementation of the virtual reality device in accordance with another aspect of the present invention.

According to FIG. 3, the virtual reality device 300 includes a memory 301 and a processor 302 for storing instructions for controlling the processor 302 to operate to perform the above-described gesture recognition method for the virtual reality device.

In addition, as shown in FIG. 3, the virtual reality device 300 further includes an interface device 303, an input device 304, a display device 305, a communication device 306, and the like. Although a plurality of devices are illustrated in FIG. 3, the present invention may relate only to some of the devices, such as the processor 301, the memory 302, the display device 305, and the like.

The communication device 306 can be wired or wirelessly communicated, for example.

The above interface device 303 includes, for example, a headphone jack, a USB interface, and the like.

The input device 304 described above may include, for example, a touch screen, a button, and the like.

The display device 305 described above is, for example, a liquid crystal display, a touch display, or the like.

The above embodiments mainly focus on the differences from the other embodiments, but it should be apparent to those skilled in the art that the above embodiments may be used alone or in combination with each other as needed.

The various embodiments in the present specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments, but the field It should be clear to the skilled person that the above embodiments can be used individually or in combination with each other as needed. In addition, for the device embodiment, since it corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the corresponding part of the method embodiment. The system embodiments described above are merely illustrative, and the modules illustrated as separate components may or may not be physically separate.

The invention can be a system, method and/or computer program product. The computer program product can include a computer readable storage medium loaded with various means for causing a processor to implement the present invention Computer readable program instructions.

The computer readable storage medium can be a tangible device that can hold and store the instructions used by the instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, with instructions stored thereon A raised structure in the hole card or groove, and any suitable combination of the above. A computer readable storage medium as used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, a light pulse through a fiber optic cable), or through a wire The electrical signal transmitted.

The computer readable program instructions described herein can be downloaded from a computer readable storage medium to various computing/processing devices or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages. Source code or object code written in any combination, including object oriented programming languages such as Smalltalk, C++, etc., as well as conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. carried out. In the case of a remote computer, the remote computer can be connected via any kind of network, including a local area network (LAN) or wide area network (WAN). To the user's computer, or, can be connected to an external computer (for example, using an Internet service provider to connect via the Internet). In some embodiments, the customized electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams can be implemented by computer readable program instructions.

The computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or a processor of other programmable data processing apparatus to produce a machine such that when executed by a processor of a computer or other programmable data processing apparatus Means for implementing the functions/acts specified in one or more of the blocks of the flowcharts and/or block diagrams. The computer readable program instructions can also be stored in a computer readable storage medium that causes the computer, programmable data processing device, and/or other device to operate in a particular manner, such that the computer readable medium storing the instructions includes An article of manufacture that includes instructions for implementing various aspects of the functions/acts recited in one or more of the flowcharts.

The computer readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on a computer, other programmable data processing device or other device to produce a computer-implemented process. Thus, instructions executed on a computer, other programmable data processing apparatus, or other device implement the functions/acts recited in one or more of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a portion of an instruction that includes one or more components for implementing the specified logical functions. Executable instructions. In some alternative implementations, the functions noted in the blocks may also occur in a different order than those illustrated in the drawings. For example, two consecutive blocks may be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented in a dedicated hardware-based system that performs the specified function or function. Or it can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Numerous modifications and changes will be apparent to those skilled in the art without departing from the scope of the invention. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements of the techniques in the <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; The scope of the invention is defined by the appended claims.

Claims

A gesture recognition method for a virtual reality device, the virtual reality device comprising a depth camera, wherein the gesture recognition method comprises:

Controlling the depth camera to capture a current hand image of the current user;

Determining, according to the current hand image, whether the current user performs a tap action, and if so, then:

Extracting a current feature from the current hand image;

Matching the current feature with a reference feature in the model, and determining a button of the current user tap based on the matching result.
The gesture recognition method according to claim 1, wherein the gesture recognition method further comprises:

Controlling the depth camera to acquire a reference hand image of the reference user;

Extracting the reference feature from the reference hand image and storing the reference feature in the model.
The gesture recognition method according to claim 1 or 2, wherein the virtual reality device further comprises a display screen, wherein the determining, before determining whether the current user is tapping according to the current image, further comprises:

A keyboard image and an initial position of the current user's finger on the keyboard image are displayed on the display screen.
The gesture recognition method according to any one of claims 1 to 3, wherein the gesture recognition method further comprises:

And displaying, according to the button pressed by the current user, a current position of the current user's finger on the keyboard image on the display screen.
A gesture recognition apparatus for a virtual reality device, comprising:

a first control module, configured to control the depth camera to collect a current hand image of the current user;

a determining module, configured to determine, according to the current hand image, whether the current user performs a tapping action;

a current feature extraction module, configured to: when the determination result of the determination module is yes, Extracting current features from the current hand image; and,

And a matching module, configured to match the current feature with a reference feature in the model, and determine, according to the matching result, the button that the current user taps.
The gesture recognition apparatus according to claim 5, wherein the gesture recognition apparatus further comprises:

a second control module, configured to control the depth camera to collect a reference hand image of the reference user;

And a reference feature extraction module, configured to extract the reference feature from the reference hand image and store the reference feature in the model.
The gesture recognition device according to claim 5 or 6, wherein the virtual reality device further comprises a display screen, wherein the gesture recognition device further comprises:

And a first display module, configured to display, on the display screen, a keyboard image and an initial position of the current user's finger on the keyboard image.
The gesture recognition apparatus according to any one of claims 5-7, wherein the gesture recognition apparatus further comprises:

And a second display module, configured to display, according to the button of the current user tap determined by the matching module, a current position of the current user's finger on the keyboard image on the display screen.
A virtual reality device, comprising the gesture recognition device according to any one of claims 5-8.
A virtual reality device, comprising: a depth camera for acquiring an image, the memory for storing an instruction, the instruction for controlling the processor to execute the right The gesture recognition method according to any one of claims 1 to 4.
A computer readable storage medium characterized by storing program code for performing the gesture recognition method according to any one of claims 1-4.