WO2018098862A1

WO2018098862A1 - Gesture recognition method and device for virtual reality apparatus, and virtual reality apparatus

Info

Publication number: WO2018098862A1
Application number: PCT/CN2016/111063
Authority: WO
Inventors: 张茜; 张绍谦; 张超
Original assignee: 歌尔科技有限公司
Priority date: 2016-11-29
Filing date: 2016-12-20
Publication date: 2018-06-07
Also published as: CN106598235A; CN106598235B

Abstract

A gesture recognition method and device for a virtual reality apparatus, and a virtual reality apparatus. The gesture recognition method comprises: controlling each camera to capture a current gesture image of a current user (S110); splicing each current gesture image to obtain a current spliced image (S120); and performing gesture recognition according to the current spliced image (S130). The gesture recognition method can enlarge photographing angles of cameras; images obtained by cameras (11, 12) at different positions are spliced by a splicing module (320) to obtain a spliced image with the photographing angle exceeding a viewing angle of a single camera, and thus, the sense of immersion of the user is improved when using a virtual reality apparatus.

Description

Gesture recognition method, device and virtual reality device for virtual reality device

Technical field

The present invention relates to the field of virtual reality device technologies, and in particular, to a gesture recognition method, apparatus, and virtual reality device for a virtual reality device.

Background technique

Virtual Reality (VR), virtual reality technology will be a key technology to support a comprehensive and integrated multidimensional information space combining qualitative and quantitative, perceptual knowledge and rational understanding. As the speed of the Internet increases, an Internet era based on virtual reality technology is quietly coming, which will dramatically change people's production and lifestyle. The specific connotation is: comprehensive use of computer graphics systems and various interfaces such as reality and control to provide immersive sensation technology in a three-dimensional environment that can be generated on a computer.

The immersion of virtual reality devices comes from isolation from the outside world, especially visual and auditory isolation, which causes the brain to be deceived and create a virtual immersion from the real world. At present, the human-computer interaction methods of virtual reality devices are mainly language recognition, eye tracking and gesture recognition.

Summary of the invention

According to a first aspect of the present invention, a gesture recognition method for a virtual reality device is provided, the virtual reality device including at least two cameras, and the gesture recognition method includes:

Controlling each of the cameras to collect a current gesture image of the current user;

Performing splicing processing on each of the current gesture images to obtain a current spliced image;

Gesture recognition is performed according to the current mosaic image.

According to a second aspect of the present invention, a gesture recognition apparatus for a virtual reality device is provided, the virtual reality device comprising at least two cameras, the gesture recognition device comprising:

a current control module, configured to control each of the cameras to collect a current gesture image of the current user;

a current splicing module, configured to perform splicing processing on each of the current gesture images to obtain a current spliced image; and,

a gesture recognition module, configured to perform gesture recognition according to the current stitched image.

According to a third aspect of the present invention, there is provided a virtual reality device comprising a processor and a memory, the memory for storing instructions for controlling the processor to perform the method according to the first aspect of the present invention Gesture recognition method.

According to a fourth aspect of the present invention, a virtual reality device is provided, comprising:

At least two cameras disposed at different positions, and the shooting angles of the cameras disposed adjacent to each other partially overlap;

a gesture recognition apparatus according to the second aspect of the present invention;

According to a fifth aspect of the invention, there is provided a computer readable storage medium storing program code for performing the gesture recognition method according to the first aspect of the invention.

Other features and advantages of the present invention will become apparent from the Detailed Description of the <RTIgt;

DRAWINGS

The accompanying drawings, which are incorporated in FIG

1 is a flow chart of an embodiment of a gesture recognition method for a virtual reality device in accordance with the present invention;

2 is a flow chart of another embodiment of a gesture recognition method for a virtual reality device according to the present invention;

3 is a block schematic diagram of an implementation structure of a gesture recognition apparatus for a virtual reality device according to the present invention;

4 is a block schematic diagram showing another implementation structure of a gesture recognition apparatus for a virtual reality device according to the present invention;

5 is a block schematic diagram of an implementation structure of a virtual reality device according to the present invention;

6 is a left side view showing another embodiment of a virtual reality device according to the present invention;

Figure 7 is a right side elevational view of another embodiment of a virtual reality device in accordance with the present invention.

Description of the reference signs:

1-camera; 11-first camera;

12-second camera.

detailed description

Various exemplary embodiments of the present invention will now be described in detail with reference to the drawings. It should be noted that the relative arrangement of the components and steps, numerical expressions and numerical values set forth in the embodiments are not intended to limit the scope of the invention unless otherwise specified.

The following description of the at least one exemplary embodiment is merely illustrative and is in no way

Techniques, methods and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but the techniques, methods and apparatus should be considered as part of the specification, where appropriate.

In all of the examples shown and discussed herein, any specific values are to be construed as illustrative only and not as a limitation. Thus, other examples of the exemplary embodiments may have different values.

It should be noted that similar reference numerals and letters indicate similar items in the following figures, and therefore, once an item is defined in one figure, it is not required to be further discussed in the subsequent figures.

In order to solve the problem that the scope of the virtual reality device gesture recognition existing in the prior art is narrow and cannot be immersed in the user, a gesture recognition method for the virtual reality device is provided, wherein the virtual reality device includes at least Two cameras, both of which can be ordinary color cameras; they can also be depth cameras; one can be a color camera and the other is a depth camera.

1 is a flow chart of an embodiment of a gesture recognition method for a virtual reality device in accordance with the present invention.

According to FIG. 1, the gesture recognition method comprises the following steps:

Step S110, controlling each camera to collect a current gesture image of the current user.

Specifically, the manner in which the camera collects the current gesture image of the current user may be, for example, collected in units of frames.

Step S120: Perform splicing processing on each current gesture image to obtain a current spliced image.

Image stitching technology is divided into image registration and image fusion. In order to splicing multiple images into one, image registration can be performed first. According to the SIFT feature points proposed by Lowe, the matching points of the images are selected and calibrated, and then all the images are registered to a coordinate system by the affine model. under. In addition to the unified coordinate system, image registration can also obtain overlapping regions of images captured by two adjacent cameras. The image fusion combines the useful information of the registered image into one picture, and at the same time processes the splicing position blur caused by the angle of view, illumination and other factors of the registered picture. Image fusion, for example, can employ Gaussian pyramid techniques.

According to FIG. 2, the step S120 may specifically include the following steps:

In step S121, each current gesture image is preprocessed to obtain a corresponding image to be registered.

The pre-processing is specifically for performing denoising, enhancement, and the like on the acquired current gesture image data, and unifying the data format, image size, and resolution.

In step S122, all the images to be registered are subjected to registration processing to obtain an image to be fused.

The image registration is specifically the alignment between each image to be registered, and the plurality of images to be registered obtained from different cameras or different times or different angles are optimally matched to obtain an image to be fused.

Image registration is always relative to multiple images. In actual work, one of the images to be registered is usually taken as the reference for registration, which is called the reference image and the other image to be registered is Search map. The general method of image registration is to first select an image sub-block centered on a target point on the reference map, and call it a template for image registration, and then let the template move in an orderly manner on the search graph. Go to a location and compare the template to the corresponding part of the search graph until the registration position is found.

The two images to be registered for the same target encountered in image registration are often obtained under different conditions, such as different imaging times, different imaging positions, and even different imaging systems, plus each in imaging. The effect of noise makes it impossible for the two images to be registered of the same target to be identical, and only to a certain degree of similarity.

According to the different methods of image matching, the image stitching algorithm can be generally divided into the following two types: based on the region-related stitching algorithm, the region-based registration method is from the image to be stitched. Starting from the gray value, the area of the same size in the reference image and the area of the same size in the reference image are calculated by least squares or other mathematical methods to calculate the difference of the gray value, and the difference is later determined to determine the overlapping area of the image to be stitched. The degree of similarity, thereby obtaining the range and position of the overlapping area of the image to be stitched, thereby achieving image stitching. It is also possible to transform the image from the time domain to the frequency domain by FFT transformation and then register. For images with a large amount of displacement, the rotation of the image can be corrected first, and then the mapping relationship between the two images can be established. Specifically, a comparison method, a layered comparison method, or a phase correlation method may be employed.

The feature-based registration method does not directly utilize the pixel values of the image, but derives the features of the image through the pixels, and then performs search matching on the corresponding feature regions of the overlapping portions of the image based on the image features. The specific method may be, for example, a ratio matching method or a feature point matching method.

Feature-based registration methods have two processes: feature extraction and feature registration. Firstly, feature points such as points, lines and regions with obvious changes in gray scale are extracted from the image to be registered to form feature sets. Then, the feature matching algorithm is used to select the feature pairs in which the corresponding relationship exists as much as possible in the feature set corresponding to the image to be registered. A series of image segmentation techniques are used for feature extraction and boundary detection. Such as canny operator, Laplacian Gaussian operator, region growth. The extracted spatial features have closed boundaries, open boundaries, intersecting lines, and other features. The algorithms of feature matching include: cross correlation, distance transformation, dynamic programming, structure matching, chain code correlation and other algorithms.

In step S123, image fusion and boundary smoothing processing are performed on the fused image to obtain a current spliced image.

The coincident regions of the image to be stitched are fused to obtain a smooth and seamless current stitched image reconstructed by stitching.

Image fusion is a process of synthesizing multiple images of the same scene obtained by multiple image sensors of different modes or multiple images of the same scene obtained by the same sensor at different times into one stitched image.

After the image registration, due to the existence of the difference between the overlapping areas of the image, if the image pixels are simply superimposed, a clear stitching seam will appear at the stitching, so it is necessary to correct the color value near the stitching seam of the image to be stitched, so that Smooth transition for seamless composition.

Commonly used fusion methods include HIS fusion method, KL transform fusion method, high-pass filter fusion method, wavelet transform fusion method, pyramid transformation fusion method, spline transformation fusion method and so on.

Step S130, performing gesture recognition according to the current mosaic image.

In this way, the angle of view of the image captured by the camera is enlarged, the range of hand movement of the user in the process of gesture recognition is expanded, the flexibility of the user is greatly improved, and the immersion of the user using VR is increased.

According to FIG. 2, the step S130 may specifically include:

Step S131, extracting a current gesture feature in the current mosaic image;

Step S132, comparing the current gesture feature with the specified gesture feature in the database.

Step S133, determining a current gesture action according to the comparison result.

The specified gesture feature may be pre-stored in the database before the virtual reality device is shipped, or may be stored in the database before the current user is used. In a specific embodiment of the present invention, the gesture recognition method further includes:

Control each camera to capture a specified gesture image of the specified user;

Performing splicing processing on each specified gesture image to obtain a reference stitching image;

The specified gesture features corresponding to the specified gestures in the stitched image are stored in the database.

If the current gesture feature is successfully matched with a specified gesture, for example, the specified gesture feature of the left slide, that is, the current gesture action is left-sliding for the specified gesture action, the response is corresponding to the function of specifying the left motion of the gesture action, for example, opening an application. Wait.

The present invention also provides a gesture recognition apparatus for a virtual reality device. 3 is a block schematic diagram of an implementation structure of a gesture recognition apparatus for a virtual reality device in accordance with the present invention.

According to FIG. 3, the gesture recognition apparatus 300 includes a current control module 310, a current splicing module 320, and a gesture recognition module 330. The current control module 310 is configured to control each camera to collect a current gesture image of the current user. The 320 is configured to perform splicing processing on each current gesture image to obtain a current spliced image. The gesture recognition module 330 is configured to perform gesture recognition according to the current spliced image.

4 is a block schematic diagram showing another implementation structure of a gesture recognition apparatus for a virtual reality device according to the present invention.

According to FIG. 4, the current splicing module 320 may further include a pre-processing unit 321, a registration unit 322, and a merging unit 323, which is used for each current gesture image. The pre-processing is performed to obtain a corresponding image to be registered. The registration unit 322 is configured to perform registration processing on all the images to be registered to obtain an image to be fused. The fusion unit 323 is configured to perform image fusion and boundary on the fused image. Smooth processing to get the current stitched image.

Further, the gesture recognition apparatus 300 may further include a feature extraction unit 331, a comparison unit 332, and an action determination unit 333, which is configured to extract a current gesture feature in the current mosaic image; the comparison unit 332 is configured to The current gesture feature is compared with a specified gesture feature in the database; the action determination unit 333 is configured to determine the current gesture action according to the comparison result.

The present invention also provides a virtual reality device, in one aspect, as shown in FIG. 5, including a processor 502 and a memory 501 for storing instructions for controlling the processor 502 to operate to perform the above A gesture recognition method for a virtual reality device.

In addition, as shown in FIG. 5, the virtual reality device 500 further includes an interface device 503, an input device 504, a display device 505, a communication device 506, and the like. Although a plurality of devices are illustrated in FIG. 5, the present invention may relate only to some of the devices, such as processor 501, memory 502, display device 505, and the like.

The communication device 506 can be wired or wirelessly communicated, for example.

The above interface device 503 includes, for example, a headphone jack, a USB interface, and the like.

The input device 504 described above may include, for example, a touch screen, a button, and the like.

The display device 505 described above is, for example, a liquid crystal display, a touch display, or the like.

The virtual reality device may be, for example, a virtual reality helmet or a virtual reality glasses or the like.

In another aspect, the virtual reality device includes at least two cameras 1 and the above-described gesture recognition device 200 for a virtual reality device for capturing a gesture image. The virtual reality device may be, for example, a virtual reality helmet or a virtual reality glasses or the like.

In a specific embodiment of the present invention, as shown in FIG. 6 and FIG. 7, the front cover of the virtual reality device is provided with four first cameras 11, and the four first cameras 11 form a front cover. The rectangle is square or square, and the viewing angles between the adjacent first cameras 11 partially overlap; the two side covers of the virtual reality device are respectively provided with a second camera 12, and each of the second cameras 12 is adjacent to the adjacent The viewing angles between the first cameras 11 partially overlap.

The four first cameras 11 can expand the horizontal and vertical angles to increase the user's hand. The range of moving downwards and left and right; the other two second cameras 12 can increase the angle of the horizontal or vertical direction, and expand the left and right movement range of the user's hand. In this way, shooting with a super-180 degree angle of view is achieved, and blind spots are avoided.

The “front cover” is specifically a side away from the user's eyes during the wearing of the virtual reality device, and the “side cover” is specifically a surface other than the “front cover” and the opposite surface of the “front cover”.

In a specific embodiment of the present invention, the cameras 1 are both depth cameras. Since the images captured by the depth camera are grayscale images, there is no step of converting the color image into a grayscale image, so that the virtual reality device performs the above. The gesture recognition method is faster, and the image captured by the depth camera has less noise.

The above embodiments mainly focus on the differences from the other embodiments, but it should be apparent to those skilled in the art that the above embodiments may be used alone or in combination with each other as needed.

The various embodiments in the present specification are described in a progressive manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments, but the field It should be clear to the skilled person that the above embodiments can be used individually or in combination with each other as needed. In addition, for the device embodiment, since it corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the corresponding part of the method embodiment. The system embodiments described above are merely illustrative, and the modules illustrated as separate components may or may not be physically separate.

The invention can be a system, method and/or computer program product. The computer program product can comprise a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement various aspects of the present invention.

The computer readable storage medium can be a tangible device that can hold and store the instructions used by the instruction execution device. The computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) Or flash memory), static random access storage (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, a punch card or a groove in the groove on which the command is stored Structure, and any suitable combination of the above. A computer readable storage medium as used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (eg, a light pulse through a fiber optic cable), or through a wire The electrical signal transmitted.

The computer readable program instructions described herein can be downloaded from a computer readable storage medium to various computing/processing devices or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages. Source code or object code written in any combination, including object oriented programming languages such as Smalltalk, C++, etc., as well as conventional procedural programming languages such as the "C" language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. carried out. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or wide area network (WAN), or can be connected to an external computer (eg, using an Internet service provider to access the Internet) connection). In some embodiments, the customized electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by utilizing state information of computer readable program instructions. Computer readable program instructions are executed to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus, and computer program products according to embodiments of the invention. It should be understood that the flow chart and / or box Each block of the figures, as well as combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer readable program instructions.

The computer readable program instructions can be provided to a general purpose computer, a special purpose computer, or a processor of other programmable data processing apparatus to produce a machine such that when executed by a processor of a computer or other programmable data processing apparatus Means for implementing the functions/acts specified in one or more of the blocks of the flowcharts and/or block diagrams. The computer readable program instructions can also be stored in a computer readable storage medium that causes the computer, programmable data processing device, and/or other device to operate in a particular manner, such that the computer readable medium storing the instructions includes An article of manufacture that includes instructions for implementing various aspects of the functions/acts recited in one or more of the flowcharts.

The computer readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to perform a series of operational steps on a computer, other programmable data processing device or other device to produce a computer-implemented process. Thus, instructions executed on a computer, other programmable data processing apparatus, or other device implement the functions/acts recited in one or more of the flowcharts and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a portion of an instruction that includes one or more components for implementing the specified logical functions. Executable instructions. In some alternative implementations, the functions noted in the blocks may also occur in a different order than those illustrated in the drawings. For example, two consecutive blocks may be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented in a dedicated hardware-based system that performs the specified function or function. Or it can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are equivalent.

The embodiments of the present invention have been described above, and the foregoing description is illustrative, not limiting, and not limited to the disclosed embodiments. Without departing from the scope of the illustrated embodiments Many modifications and variations will be apparent to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or technical improvements of the techniques in the <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; The scope of the invention is defined by the appended claims.

Claims

A gesture recognition method for a virtual reality device, the virtual reality device comprising at least two cameras, wherein the gesture recognition method comprises:

Controlling each of the cameras to collect a current gesture image of the current user;

Performing splicing processing on each of the current gesture images to obtain a current spliced image;

Gesture recognition is performed according to the current mosaic image.
The gesture recognition method according to claim 1, wherein the splicing process of each of the current gesture images to obtain the current spliced image comprises:

Performing pre-processing on each of the current gesture images to obtain a corresponding image to be registered;

Performing registration processing on all the to-be-registered images to obtain an image to be fused;

Image fusion and boundary smoothing processing are performed on the image to be fused to obtain the current spliced image.
The gesture recognition method according to claim 1 or 2, wherein the gesture recognition according to the current mosaic image is specifically:

Extracting a current gesture feature in the current mosaic image;

Comparing the current gesture feature with a specified gesture feature in the database;

The current gesture action is determined based on the comparison result.
A gesture recognition device for a virtual reality device, the virtual reality device comprising at least two cameras, wherein the gesture recognition device comprises:

a current control module, configured to control each of the cameras to collect a current gesture image of the current user;

a current splicing module, configured to perform splicing processing on each of the current gesture images to obtain a current spliced image; and,

a gesture recognition module, configured to perform gesture recognition according to the current stitched image.
The gesture recognition apparatus according to claim 4, wherein the current splicing module specifically includes:

a pre-processing unit, configured to perform pre-processing on each of the current gesture images to obtain a corresponding image to be registered;

a registration unit, configured to perform registration processing on all the to-be-registered images to obtain a to-be-fused image image;

The merging unit is configured to perform image fusion and boundary smoothing processing on the image to be fused to obtain the current spliced image.
The gesture recognition apparatus according to claim 4 or 5, wherein the gesture recognition module further comprises:

a feature extraction unit, configured to extract a current gesture feature in the current mosaic image;

a comparison unit, configured to compare the current gesture feature with a specified gesture feature in a database;

And an action determining unit, configured to determine a current gesture action according to the comparison result.
A virtual reality device, comprising a processor and a memory, the memory for storing instructions for controlling the processor to perform the gesture of any one of claims 1-3 recognition methods.
A virtual reality device, comprising:

At least two cameras disposed at different positions, and the shooting angles of the cameras disposed adjacent to each other partially overlap;

a gesture recognition apparatus according to any one of claims 4-6;
The virtual reality device according to claim 8, wherein the front cover of the virtual reality device is provided with four first cameras, and the viewing angles between adjacent first cameras partially overlap; the virtual reality A second camera is disposed on each of the two side covers of the device, and a viewing angle between each of the second cameras and the adjacent first camera partially overlaps.
A virtual reality device according to claim 8 or 9, wherein each of said cameras is a depth camera.
A computer readable storage medium characterized by storing program code for performing the gesture recognition method according to any one of claims 1-3.