CN106325509A - Method and system for three-dimensional gesture recognition - Google Patents

Method and system for three-dimensional gesture recognition Download PDF

Info

Publication number
CN106325509A
CN106325509A CN201610694390.6A CN201610694390A CN106325509A CN 106325509 A CN106325509 A CN 106325509A CN 201610694390 A CN201610694390 A CN 201610694390A CN 106325509 A CN106325509 A CN 106325509A
Authority
CN
China
Prior art keywords
hand
user
dimensional
depth
information
Prior art date
Application number
CN201610694390.6A
Other languages
Chinese (zh)
Inventor
伊威
古鉴
方维
杨婷
马宝庆
Original Assignee
北京暴风魔镜科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京暴风魔镜科技有限公司 filed Critical 北京暴风魔镜科技有限公司
Priority to CN201610694390.6A priority Critical patent/CN106325509A/en
Publication of CN106325509A publication Critical patent/CN106325509A/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017Gesture based interaction, e.g. based on a set of recognized hand gestures
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06KRECOGNITION OF DATA; PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS
    • G06K9/00Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints
    • G06K9/00362Recognising human body or animal bodies, e.g. vehicle occupant, pedestrian; Recognising body parts, e.g. hand
    • G06K9/00375Recognition of hand or arm, e.g. static hand biometric or posture recognition
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06NCOMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computer systems based on biological models
    • G06N3/02Computer systems based on biological models using neural network models
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/006Mixed reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00Manipulating 3D models or images for computer graphics
    • G06T19/20Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts

Abstract

The invention discloses a three-dimensional gesture recognition method and system. The three-dimensional gesture recognition method comprises the steps: acquiring first three-dimensional position information on a hand of a user, wherein the first three-dimensional position information is position information on a first position point of the hand of the user; carrying out predicting calculation on the first three-dimensional position information by using a predefined gesture prediction algorithm so as to obtain second three-dimensional position information and attitude information on the hand of the user, wherein the second three-dimensional position information is position information on a second position point of the hand of the user; inputting the second three-dimensional position information and the attitude information to a pre-modeled three-dimensional hand model, thereby obtaining a three-dimensional gesture corresponding to the hand of the user. According to the technical scheme disclosed by embodiments of the invention, the simulation of real actions of hands in the fields of virtual reality and augment reality is achieved, and thus the effects of recognizing and interacting three-dimensional gestures are achieved.

Description

三维手势识别方法及系统 Method and system for three-dimensional gesture recognition

技术领域 FIELD

[0001 ]本公开一般涉及计算机技术领域,具体涉及虚拟现实和增强现实技术领域,尤其涉及一种三维手势识别方法及系统。 [0001] The present disclosure generally relates to computer technologies, and particularly relates to virtual reality and augmented reality technology, and particularly relates to a three-dimensional gesture recognition method and system.

背景技术 Background technique

[0002]手势交互能在不同场景下为用户提供自然交互的可能,其广泛应用于游戏、房地产、教育、旅游、影视等众多领域,用户无需穿戴任何设备,便可以实现如同人手跟自然世界一样的交互动作。 [0002] gesture interaction can provide users in different scenarios for the possible natural interaction, which is widely used in many areas of the game, real estate, education, tourism, film and television, users do not need to wear any equipment, we can achieve the same as manpower with the natural world interactive action. 同时,该技术是虚拟现实和增强现实应用领域中最为关键的人机交互技术之一,是实现更好的交互体验或更为复杂功能的基础。 At the same time, the technology is virtual reality and augmented reality applications in one of the most critical human-computer interaction technology, better interactive experience or base is more complex functions. 通过手势交互技术,可以极大地增强用户在使用虚拟现实(VR)/增强现实(AR)设备时的真实感和沉浸感。 Interactive technology through gestures, users can greatly enhance the use of virtual reality (VR) / Augmented Reality (AR) reality and immersion time of the device. 当前,精准捕捉、低延时、低功耗、便于携带、低成本的手势交互系统是该领域研究发展的重点方向。 Current, accurate capture, low-latency, low-power, portable, low-cost gesture interaction system is the focus of the direction of research and development in this field.

[0003]从交互上看,手势作为是一种输入模式,其通过相关的外部设备获取模拟手部动作的输出。 [0003] From the point of view interact, as a gesture input mode that acquires output analog hand motions through the relevant external device. 人机交互是指人与机器之间的互动方式,这种互动方式经历了鼠标、物理硬件、屏幕触控、远距离的体感操作的逐步发展的过程。 Human-computer interaction refers to the interaction between man and machine, this interactive experience mouse, the physical hardware, touch screen, the process of gradual development of somatosensory remote operation. 传统的手势交互方式具体如下: The traditional gesture interaction as follows:

[0004] I)利用鼠标、光标的轨迹模拟手势交互。 [0004] I) using the mouse, the cursor trajectory simulation gesture interaction. 通过手部握住鼠标在显示屏上下左右滑动,来近似模拟手部方位运动。 Mouse slide up and down around the display screen, to approximate the orientation of the hand movement by the hand grip portion. 该方案缺点为,鼠标的动作非常单一,只有二维而没有三维信息,无法模拟手部的真实动作。 The program is a drawback, mouse movements are very simple, only two-dimensional and three-dimensional information is not, can not simulate the real movement of the hand.

[0005] 2)使用触摸板进行单指或多指等多种手势交互。 [0005] 2) using a single touchpad or more fingers and other gestures to interact. 例如,使用笔记本外置的触摸板设备,通过单指或多指的滑动,近似模拟手部的方位运动。 E.g., using a laptop touchpad external device, by a single or more fingers sliding orientation approximation analog hand movement. 这类方法与鼠标光标的手势交互相同,无法模拟手部的真实动作。 Such methods and the same mouse cursor gesture interaction, can not simulate the real movement of the hand.

[0006] 3)触摸屏上的手势交互。 [0006] 3) on the touch screen gesture interaction. 移动端(平板、手机)使用触摸屏的手势交互,主要有长按、轻触、滑动、拖动、旋转、缩放、摇动这八种手势,其优点是增加了可便携性,简单模拟了手势交互动作,其缺点是手势交互动作过于单一,无法模拟手部的真实动作。 Mobile terminal (plate, mobile phone) using a touch screen gesture interaction, mainly press, touch, a slide, drag, rotate, zoom, pan the eight gestures, the advantage of increased portability, a simple simulation gesture interaction action, the drawback is the interaction over a single gesture, you can not simulate the real movement of the hand.

[0007]由此可见,目前的手势交互方式大部分无法完全模拟手部的真实动作,而且无法应用在虚拟现实和增强现实领域中。 [0007] Thus, the current gesture interaction majority can not fully simulate the real movement of the hand and can not be used in virtual reality and augmented reality in the field. 然而,针对此问题,现有技术并没有提供一种有效的解决方案。 However, for this problem, the prior art does not provide an effective solution.

发明内容 SUMMARY

[0008]鉴于现有技术中的上述缺陷或不足,期望提供一种能够在虚拟现实和增强现实领域中模拟手部真实动作,从而实现便携式智能移动设备与虚拟现实/增强现实设备之间的手势交互实的技术方案。 [0008] In view of the above prior art defects or deficiencies, it is desirable to provide a true simulation of the operation of the hand in the field of virtual reality and augmented reality in order to achieve a portable smart mobile devices and virtual reality / enhanced reality devices between gesture interactive real technical solutions.

[0009]第一方面,本申请提供了一种三维手势识别方法,所述方法包括:获取用户手部的第一三维位置信息,所述第一三维位置信息是用户手部上第一位置点的位置信息;使用预先定义的手势预测算法对所述第一三维位置信息进行预测计算,得到用户手部的第二三维位置信息和姿态信息,所述第二三维位置信息是用户手部上第二位置点的位置信息;以及将所述第二三维位置信息和所述姿态信息输入预先构建的三维手部模型,得到用户手部对应的三维手势。 [0009] In a first aspect, the present application provides a three-dimensional gesture recognition, the method comprising: obtaining a first three-dimensional position information of the user's hand, the first three-dimensional position information on the user's hand portion is a first position of a point position information; use the predefined gesture prediction algorithm to the first three-dimensional position information prediction calculation to obtain a second dimensional position information and attitude information of the user's hand, a second three-dimensional position information of the first user's hand is position information of the position of two points; and the three-dimensional model of the hand of the second three-dimensional position information and posture information input to the pre-built, to give the corresponding three-dimensional user's hand gestures.

[0010]第二方面,本申请提供了一种三维手势识别系统,包括穿戴式设备和终端设备,所述穿戴式设备包括:用于置于用户头部的头部固定结构;深度传感器,用于获取用户手部及周围环境的深度图像信息;接口传输结构,设置在所述深度传感器上,通过所述接口传输结构,所述深度传感器能够将所述深度图像信息发送给能够拆卸式安装在所述穿戴式设备上的终端设备;所述终端设备包括:提取模块,用于根据预置的手部形状和深度特征值,从所述深度图像信息中提取出所述第一三维位置信息;计算模块,用于使用预先定义的手势预测算法对所述第一三维位置信息进行预测计算,得到用户手部的第二三维位置信息和姿态信息,所述第二三维位置信息是用户手部上第二位置点的位置信息;处理模块,用于将所述第二三维位置信息和所述姿态信息输入预先构建的三 [0010] In a second aspect, the present application provides a three-dimensional gesture recognition system, comprising a wearable device and a terminal device, the wearable device comprising: a fixing structure for a head-mounted head of a user; a depth sensor, with acquiring a depth image in the user's hand and ambient environment information; interface transfer structure disposed on the depth sensor, transmitted via the interface structure, the depth of the depth image sensor capable of transmitting information to be detachably mounted on the wearable terminals on the device; and the terminal device comprising: an extracting module, according to hand shape and depth of a preset characteristic value is extracted from the first three-dimensional position information of the depth image information; calculation module using a predefined gesture for the first three-dimensional prediction algorithm to predict the position information calculation, the second three-dimensional position information and attitude information of the user's hand, a second three-dimensional position information on the user's hand is position information of the second location point; processing module, for the second three-dimensional position information and posture information input to the pre-built three 维手部模型,得到用户手部对应的三维手势。 Hand-dimensional model corresponding to the user's hand to give three-dimensional gesture.

[0011]根据本申请实施例提供的技术方案,通过先由深度传感器获取用户手部的三维位置信息,再由终端设备根据预先设置的手势预测算法和预先构建的三维手部模型对用户手部的三维位置信息进行处理,从而得到用户手部对应的三维手势,最终达到了在虚拟现实和增强现实领域中模拟手部真实动作的效果。 [0011] According to the embodiment provided in the present application, three-dimensional position information acquired by the user's hand first by a depth sensor, then the prediction algorithm by the terminal device and the three-dimensional model of the hand gesture previously constructed according to a preset user's hand the three-dimensional position information is processed to obtain the corresponding three-dimensional user's hand gesture, and ultimately achieve the effect of simulated hand in a real operation of virtual reality and augmented reality art.

附图说明 BRIEF DESCRIPTION

[0012]通过阅读参照以下附图所作的对非限制性实施例所作的详细描述,本申请的其它特征、目的和优点将会变得更明显: [0012] By reading the following detailed description of the accompanying drawings of non-limiting embodiments, and other features, objects and advantages of the present disclosure will become more apparent:

[0013]图1是根据本申请的三维手势识别方法流程图; [0013] FIG. 1 is a flowchart illustrating a three-dimensional gesture recognition method of the present disclosure;

[0014]图2A是根据本申请的三维手势识别系统的结构示意图; [0014] FIG. 2A is a schematic structural diagram of a three-dimensional gesture recognition system of the present application;

[0015]图2B是根据本申请的三维手势识别系统中终端设备的结构框图; [0015] FIG. 2B is a block diagram showing a three-dimensional gesture recognition system of the present application, the terminal device;

[0016]图3是根据本申请的三维手势交互过程示意图;以及 [0016] FIG. 3 is a schematic diagram of a three-dimensional gesture interaction process of the present application; and

[0017]图4是根据本申请的使用VR/AR设备与终端设备进行三维手势交互的效果示意图。 [0017] FIG. 4 is a schematic diagram of a three-dimensional gesture interaction effects according to the use of the present disclosure VR / AR device and the terminal device.

具体实施方式 Detailed ways

[0018]下面结合附图和实施例对本申请作进一步的详细说明。 Drawings and embodiments of the present application will be further described in detail [0018] below in conjunction. 可以理解的是,此处所描述的具体实施例仅仅用于解释相关发明,而非对该发明的限定。 It will be appreciated that the specific embodiments described herein are only relevant for explaining the invention, not limitation of the invention. 另外还需要说明的是,为了便于描述,附图中仅示出了与发明相关的部分。 Also to be noted also that, for convenience of description, the accompanying drawings show only parts related to the invention.

[0019]需要说明的是,在不冲突的情况下,本申请中的实施例及实施例中的特征可以相互组合。 [0019] Incidentally, in the case of no conflict, embodiments and features of the embodiments of the present application can be combined with each other. 下面将参考附图并结合实施例来详细说明本申请。 Below with reference to the accompanying drawings and described in detail in conjunction with embodiments of the present application.

[0020]相较于传统的几种手势交互方式,现有技术中也存在可以实现真实模拟手部动作的手势交互方式,例如:方式I,通过在手部固定至少一个传感器设备,从而对手部的动作进行捕捉,这种手势交互方式能够真实模拟手部的动作,但是其严重依赖外部的传感器设备,成本高、体积大,便携性差,更为关键的是,还需要在用户手部上固定传感器,这给用户操作带来不好的体验。 [0020] Compared with the conventional gesture interaction of several ways, the prior art may also be present gesture interaction provides a realistic simulation of the operation of the hand, for example: Mode I, by the hand portion securing at least one sensor device, so that the opponent portion capture operation, such a gesture can be interaction simulate real operation hand, but it depends heavily on external sensor device, the high cost, bulky, poor portability, more critical, also needs to be fixed on the user's hand sensor, which brings bad experience to the user. 方式2,通过使用双目摄像头或者深度摄像头获取手部的三维信息,将手部的三维姿态重建出来,从而模拟真实的手部动作,虽然这种手势交互方式无需在手部增加额外的传感器设备,但是这种方式需要结合PC才能完成,这是由于模拟过程的算法过于复杂对处理芯片的要求过高,因此严重依赖于PC的硬件性能,导致其无法集成在注重便携性的智能移动设备上来实现。 Embodiment 2, by using a binocular camera or depth camera acquiring three-dimensional information of the hand, the three-dimensional posture of the hand portion of the reconstruction out, to simulate a real hand movements, although this gesture interaction without additional sensor device in the hand , but this approach requires a combination of PC to complete, this is because the algorithm simulation process is too complicated excessive demands on the processor chip, and therefore heavily dependent on PC hardware performance, leading to its emphasis on portability can not be integrated in smart mobile devices up achieve.

[0021]可以看出,这两种方式虽然可以模拟用户手部的动作,但是其由于各自的缺陷导致无法应用在日渐成熟的虚拟现实和增强现实技术领域,从而无法为用户提供一种较好用户体验的三维手势交互方案。 [0021] As can be seen, although these two methods can simulate the motion of the user's hand, but due to the respective defects result can not be applied in the maturing of virtual reality and augmented reality technology areas, making it impossible to provide users with a better user experience three-dimensional gesture interaction program.

[0022]而本申请提供的实施例提供的技术方案着重从虚拟现实和增强现实领域,提出一种可以实现虚拟现实/增强现实等头戴式设备与智能移动设备之间的三维手势交互方案,整个交互过程对硬件性能大大降低,只需要在头戴式设备上增设处理芯片或者采用智能移动设备自带的处理芯片,即可完成整个三维手势交互过程。 [0022] and the technical solutions provided in embodiments provided herein focuses on the field of virtual reality and augmented reality, virtual reality propose a / enhanced three-dimensional gesture interaction between programs reality headsets and smart mobile devices, the entire interaction process is greatly reduced performance of the hardware, only the need for additional processing chip on the head mounted device or the mobile device comes with a smart chip, to complete the entire three-dimensional gesture interaction.

[0023]请参考图1,图1是根据本申请的三维手势识别方法流程图,如图1所示,该流程包括以下步骤(步骤S102-步骤S106): [0023] Please refer to FIG 1, FIG. 1 is a three-dimensional gesture recognition method of the present application, a flow chart shown in Figure 1, which comprises the following steps (step S102- Step S106):

[0024]步骤S102、获取用户手部的第一三维位置信息,所述第一三维位置信息是用户手部上第一位置点的位置信息; [0024] In step S102, obtaining a first three-dimensional position information of the user's hand, the first three-dimensional position information is position information of the first user's hand position of the point;

[0025]步骤S104、使用预先定义的手势预测算法对所述第一三维位置信息进行预测计算,得到用户手部的第二三维位置信息和姿态信息,所述第二三维位置信息是用户手部上第二位置点的位置信息;以及 [0025] step S104, using a predefined gesture prediction algorithm to the first three-dimensional position information prediction calculation to obtain a second dimensional position information and attitude information of the user's hand, a second three-dimensional position information of the user's hand is a second location point position information; and

[0026]步骤S106、将所述第二三维位置信息和所述姿态信息输入预先构建的三维手部模型,得到用户手部对应的三维手势。 [0026] step S106, the three-dimensional model of a second three-dimensional hand position information and the posture information input pre-built, to give the corresponding three-dimensional user's hand gestures.

[0027]通过上述步骤,可以实现虚拟现实/增强现实等头戴式设备与智能移动设备之间的三维手势交互方案,整个交互过程对硬件性能大大降低。 [0027] Through the above steps, the virtual reality / enhanced three-dimensional gesture interaction between programs reality headsets and smart mobile devices, the entire interaction process greatly reduces the hardware performance.

[0028]在上述步骤S102,对于获取用户手部的第一三维位置信息的过程,可以通过这样的方式来实现:先通过深度传感器获取用户手部及周围环境的深度图像信息,再根据预置的手部形状和深度特征值,从所述深度图像信息中提取出所述第一三维位置信息。 [0028] In the step S102, the information for obtaining a first three-dimensional position of the user's hand process, may be achieved by way of: obtaining the user's hand and the first image information of the surroundings of the depth by a depth sensor, and then according to a preset hand shape and depth of the feature value is extracted from the first three-dimensional position information of the depth image information.

[0029]作为一个较佳的实现方式,深度传感器上的芯片或处理模块只负责采集深度图像信息,然后可以将该深度图像信息发送给智能手机等终端设备,借助终端设备上的处理芯片(如CPU或GPU)的强大处理功能,负责根据预置的手部形状和深度特征值,从所述深度图像信息中提取出所述第一三维位置信息。 [0029] As a preferred implementation, the processing module on a chip or a depth sensor is only responsible for collecting information of the depth image, the depth image may then transmit the information to the smart phone and other terminal equipment, means of processing chip on the terminal device (e.g. CPU or GPU) powerful processing capabilities, are responsible for hand shape and depth of a preset characteristic value is extracted from the first three-dimensional position information of the depth image information.

[0030]在本申请实施例中,所述第一位置点为用户手部的轮廓点,所述第二位置点为用户手部的关节点,所述姿态信息为用户手部的骨骼之间的角度。 [0030] In the present application embodiment, the position of the first point of the contour point of the user's hand, said second articular point position of the point of the user's hand, for the bone between the user's hand posture information Angle.

[0031]也就是说,先借助深度传感器获取其图像采集范围(当然包含用户手部)的深度图像,在实际应用中,可以预先采集多个手部样本数据而得到用户手部的形状,但是总体而言,用户手部基本都是包含一个手掌部和五个手指的形状,而且,手的各个部位(例如,手掌部边缘、五个指尖)等对应到深度传感器的特征值是不同的,不同的深度特征值可以作为区分可各个部位的空间位置,因此,再根据预先设置的手部形状及深度特征值就可以将手从深度图中分割出来,得到手的大概轮廓,并进一步确定大概轮廓上预置点(即所述第一位置点)的位置信息,也即上述第一三维位置信息。 [0031] That is, the depth of the first image sensor acquires a depth which means the image capture range (of course, contain the user's hand), and in practical applications, can be collected in advance a plurality of sample data obtained hand shape of the user's hand, but the in general, the user's hand basically comprises a palm portion and a shape of the five fingers, and the respective parts of the hand (e.g., palm edge, five fingertip) or the like corresponding to the value of the depth characteristic is different sensors , different depth characteristic value may be used as discriminating the spatial position of each part, therefore, then it can be hand segmented according to the hand shape and depth characteristic value set in advance from the depth map out, to give a hand approximate outline, and further determines probably the preset profile (i.e., point to the first position) of the position information, i.e., the first three-dimensional position information.

[0032]在实际应用中,可以将深度传感器设置在头戴式设备的前面,使用户手部位于其图像采集范围内,当然,对于目前的VR设备来说,很多采用智能手机等移动设备作为其场景提供设备,由于考虑到制作成本和技术成熟度,目前的移动设备大多采用普通的摄像头而非深度传感器,但随着智能技术的发展,未来的智能移动设备很大可能就会采用深度传感器,如果这样,头戴式VR设备上也可以不设置深度传感器,而直接利用智能移动设备上自带的深度传感器。 [0032] In practical applications, the sensor may be disposed in front of the depth of the head-mounted device, so that the user's hand within the image capture range portion thereof, of course, for the current VR devices, many mobile devices such as smart phones as its scenes provision of equipment, taking into consideration production costs and technology maturity, current mobile devices they use ordinary camera rather than the depth sensor, but with the development of smart technology, the future of smart mobile devices will very likely adopt depth sensor If so, a depth sensor may not be provided on the head mounted device VR, directly using its own smart mobile devices depth sensor.

[0033]也就是说,深度传感器设置在头戴式VR设备上还是设置在智能移动设备上都是可行的技术方案。 [0033] That is, the depth sensor is disposed is provided on the smart mobile devices are feasible aspect head mounted on the VR device.

[0034]其中,对于深度传感器的种类并不作出限定,例如,本申请中,所述深度传感器可以采用结构光相机、也可以采用飞行时间(Time of Flight,简称为T0F)相机。 [0034] wherein the depth of the kind made by the sensor is not defined, e.g., in the present application, the depth of the sensor camera may employ structured light, time of flight may be used (Time of Flight, simply referred to as the T0F) camera.

[0035]在本申请实施例中,所述手势预测算法是根据预先定义的深度学习算法对多个深度训练数据进行学习后得到的深度训练模型。 [0035] In the embodiment of the present application, the gesture trained predictive model algorithm is a depth of a plurality of learning data based on the depth of the depth of training predefined learning algorithm obtained. 在得到上述第一三维位置信息之后,就可以根据手势预测算法,获取手部的姿态信息及关键点(用户手部上第二位置点,例如手指上的各个关节点)的位置信息(即所述第二三维位置信息),最后将这两种信息输入到预先构建的三维手部模型,从而驱动三维手部模型,输出与当前用户手部对应的三维手势。 After obtaining the first three-dimensional position information, the prediction algorithm may according to the gesture, posture information acquisition key and the hand portion (a second position of the point on the user's hand, for example, the respective finger joints) location information (i.e., the said second three-dimensional position information), these two last information is input to the three-dimensional model constructed in advance of the hand, the hand unit to drive the three-dimensional model, and the output current corresponding to the three-dimensional portion of the user's hand gestures.

[0036]对于传统VR/AR设备而言,由于用户手部的三维手势与用户所要执行的操作指令是存在预设的对应关系的,例如,手指的拿捏动作代表着拉大虚拟显示画面,手指的单指点击动作代表着打开画面内容等等。 [0036] For the conventional VR / AR device, since the three-dimensional gesture operation instruction of the user the user's hand to be executed is the presence of a predetermined correspondence relationship, e.g., just the right finger movements represent widening virtual display screen, the finger the action represents a single finger tap to open the contents of the screen, and so on.

[0037]因此,只要识别出当前用户手部对应的三维手势,也就相当于获得了当前用户手部所要表达的操作指令。 [0037] Thus, as long as the current user to identify a hand gesture corresponding to a three-dimensional, it is equivalent to the operation instruction is obtained to express the current user's hand. 在实际应用中,只要设置了处理能力的设备(如智能手机)对操作指令进行分析并执行,即可实现用户与VR/AR设备之间进行交互的目的。 In practical applications, as long as the processing capability of the device (e.g. Smartphone) is provided to analyze and execute the operational instructions, to achieve the purpose of interaction between the user and VR / AR device.

[0038]对应于上述三维手势识别方法,本申请实施例还提供了一种三维手势识别系统,如图2A(图2A是根据本申请的三维手势识别系统的结构示意图)所示,该三维手势识别系统包括穿戴式设备I和终端设备2,其中: [0038] corresponding to the three-dimensional gesture recognition method, the present application further provides a three-dimensional gesture recognition system, shown in FIG 2A (FIG. 2A is a schematic three-dimensional configuration of the gesture recognition system according to the present application), this three-dimensional gesture identification system comprising a wearable device I and a terminal device 2, wherein:

[0039]所述穿戴式设备I包括: [0039] I said wearable device comprising:

[0040]用于置于用户头部的头部固定结构11; [0040] The fixing structure for a head-mounted head of the user 11;

[0041]深度传感器12,用于获取用户手部及周围环境的深度图像信息; [0041] The depth sensor 12, and the user's hand for acquiring depth image information of the surrounding environment;

[0042]接口传输结构13,设置在所述深度传感器12上,通过所述接口传输结构13,所述深度传感器12能够将所述深度图像信息发送给能够拆卸式安装在所述穿戴式设备I上的终端设备2 ; [0042] interface transfer structure 13, disposed on the depth sensor 12, the transmission through the interface structure 13, the depth sensor 12 can send a message to the depth image to be detachably mountable to the wearable device I 2 on the terminal apparatus;

[0043]请同时参考图2B,图2B是根据本申请的三维手势识别系统中终端设备的结构框图,如图2B所示,所述终端设备2可以进一步包括: [0043] Please refer to FIG. 2B, FIG 2B is a block diagram showing a three-dimensional gesture recognition system of the present application, the terminal apparatus according to FIG. 2B, the terminal apparatus 2 may further comprise:

[0044]提取模块21,用于根据预置的手部形状和深度特征值,从所述深度图像信息中提取出所述第一三维位置信息; [0044] The extraction module 21, according to hand shape and depth of a preset characteristic value is extracted from the first three-dimensional position information of the depth image information;

[0045]计算模块22,用于使用预先定义的手势预测算法对所述第一三维位置信息进行预测计算,得到用户手部的第二三维位置信息和姿态信息,所述第二三维位置信息是用户手部上第二位置点的位置信息; [0045] The calculation module 22, using a predefined gesture prediction algorithm to the first three-dimensional position information prediction calculation to obtain a second dimensional position information and attitude information of the user's hand, a second three-dimensional position information position information on the position of the second point of the user's hand;

[0046]处理模块23,用于将所述第二三维位置信息和所述姿态信息输入预先构建的三维手部模型,得到用户手部对应的三维手势。 [0046] The processing module 23, a three-dimensional model of the second three-dimensional hand position information and posture information input to the pre-built, to give the corresponding three-dimensional user's hand gestures.

[0047]本申请实施例中,所述手势预测算法是根据预先定义的深度学习算法对多个深度训练数据进行学习后得到的深度训练模型。 [0047] Example embodiments of the present application, the gesture prediction algorithm based on the depth is the depth of a training model learning algorithm obtained after a predefined depth of the plurality of training data for learning. 所述第一位置点为用户手部的轮廓点,所述第二位置点为用户手部的关节点,所述姿态信息为用户手部的骨骼之间的角度。 The first point of the contour point position of the user's hand, said second articular point position of the point of the user's hand, the posture information as the angle between the bones of the user's hand.

[0048]本申请实施例中,所述深度传感器可以采用结构光相机、也可以采用飞行时间(Time of Flight,简称为TOF)相机。 [0048] Application of the present embodiment, the depth sensor may employ structured light camera, a time of flight may be used (Time of Flight, simply referred to as TOF) camera. 当然,对于深度传感器的种类并不作出限定,实际应用中,还可以采用其他深度传感器或者其它具有类似效果的传感器。 Of course, the kind made by the depth sensor is not defined, the practical application, but also other depth sensor or other sensors may be employed with similar effect.

[0049]在三维手势识别系统的工作流程中,对于所述深度传感器,其可以获取精确的深度图数据,其可以采用结构光相机,结构光相机可以采用成熟的CMOS传感器,应用了特殊红外波段打光,加入了对应波段的红外窄带带通滤光片,通过双目摄像头的标定之后,结合特征匹配计算深度值,也可以采用TOF相机,TOF相机是一个激光正面,通过发射和接收光信号的相位差,直接算出深度值。 [0049] In the three-dimensional gesture recognition workflow system, with respect to the depth sensor, which may acquire accurate depth map data, which may employ structured light camera, a structured light camera mature CMOS sensor can be used, applied special infrared lighting, the addition of an infrared wavelength band corresponding to the narrow band pass filter, after calibration by binocular camera, a depth value matching calculation binding characteristics, may be used TOF camera, a laser TOF camera is positive, by transmitting and receiving light signals phase difference, calculating a depth value directly.

[0050]本申请实施例中,采用一个深度摄像头与智能移动设备(即上述终端设备)固定并相连接,提供深度点云数据,智能移动设备读取数据,并进行实时的手部姿态和位置的估 [0050] The present application example using a depth camera and smart mobile devices (i.e., the terminal apparatus) is fixed and connected to provide depth point cloud data, intelligent mobile device to read the data, and real-time hand attitude and position estimates

i+o i + o

[0051]对于所述手势预测算法,在设定首先由人工标注大量手部的不同视角不同姿态的基于深度的训练数据,然后利用深度学习算法训练数据,得到一个深度训练模型(即为所述手势预测算法),通过该手势预测算法,即可输出手部骨骼的姿态信息(即所述姿态信息)和关节点的三维位置信息(即所述第二三维位置信息)。 [0051] For the gesture prediction algorithm, the first set of training data based on the depth of a large number of different perspectives manual annotation hand posture different, then the training data using the depth learning algorithm, training to obtain a depth model (ie the gesture prediction algorithm) algorithm, to output the hand bones posture information (i.e., the posture information) and three-dimensional position information of the joints (i.e., the second three-dimensional position information) predicted by the gesture.

[0052]另外,还为用户手部虚拟出一个三维手部模型,在使用过程中,将真实数据(即实时获取到的姿态信息和第二三维位置信息)输入三维手部模型,即可得到用户手部的三维手势,进而产生与真实世界一直的手部动作,从而确定用户的手部动作对应的操作指令。 [0052] In addition, the user's hand to a three-dimensional virtual model of the hand, during use, the actual data (i.e., real-time access to information and the posture information of the second three-dimensional position) input three-dimensional model of the hand, to obtain three-dimensional user's hand gesture, hand movements and produce the real world has been to determine the operation of the hand operation command corresponding to the user.

[0053]本申请实施例中,智能移动设备等终端设备的功能除了为手势预测算法提供CPU/GHJ计算支持以完成上述计算和处理操作之外,还要产生VR/AR的场景内容。 [0053] Example embodiment of the present application, mobile device, smart function terminal device in addition to providing CPU / GHJ gesture prediction algorithm is to support the above calculations and processing operations, should also produce scene content VR / AR's.

[0054]对于所述穿戴式设备,例如VR/AR头戴式设备等,主要用于借助智能移动设备等提供的二维视频画面生成全景的视频画面,增强了虚拟现实和增强现实应用的沉浸感体验。 [0054] For the wearable device, e.g. VR / AR headsets, mainly by means of a two-dimensional video picture to provide a smart mobile device generating a panoramic video picture, the enhanced virtual reality and augmented reality application immersion sense experience.

[0055]进一步地,VR/AR应用场景部分,可以将交互技术与全景视频相结合,进一步增强了虚拟现实和增强现实应用的沉浸感体验。 [0055] Further, VR / AR scenarios section, you can interact with the panoramic video technology combine to further enhance the immersive virtual reality and augmented reality applications experience.

[0056]在使用过程中,由智能移动设备接入VR/AR头显,生成全景的实时视频内容,使用前面提到的虚拟手模型,完成在VR/AR应用场景中的交互。 [0056] During use, AR head considerably by an intelligent mobile device to access VR /, generating a panoramic real-time video content, using the virtual model of the hand previously mentioned, completed in VR / AR interactive application scenarios.

[0057]为进一步理解三维手势识别系统中各部部分的工作过程,可以参考附图3(图3是根据本申请的三维手势交互过程示意图),由于各部分的工作原理已经在前面进行了介绍,此处不在结合附图3进行进一步的说明。 [0057] For a further understanding the three-dimensional working process of the gesture recognition system in each part section, reference may be 3 (FIG. 3 is a schematic three-dimensional gesture interaction process of the present application) the drawings, since the working principle of each part have been described in the foregoing, 3 here not be further described in conjunction with the accompanying drawings.

[0058]为便于理解三维手势识别系统中穿戴式设备与终端设备之间进行交互的过程,以及呈现在用户眼前的虚拟现实效果,可以参考图4(图4是根据本申请的使用VR/AR设备与终端设备进行三维手势交互的效果示意图),以下对用户使用三维手势识别系统的过程进行简单介绍: [0058] To facilitate understanding of the process of interaction between the three-dimensional gesture recognition system and the terminal device wearable devices, and virtual reality effects appear to users and to be reference to Figure 4 (FIG. 4 is a use of the present disclosure VR / AR device and terminal device schematic three-dimensional gesture interaction effect), the following three-dimensional gesture recognition system to process user briefly:

[0059]首先,将深度摄像头固定在VR/AR头戴式显示设备上,然后将智能移动设备(如智能手机)挂载或嵌入VR/AR头戴式显示设备中,进行固定,并用数据线将深度摄像头与智能手机相连接。 [0059] First, the fixed depth camera VR / AR head mounted display device, then the smart mobile devices (e.g. smart phones) mounted or embedded VR / AR head mounted display device, fixed, and data lines the depth camera is connected to the smartphone. 从智能手机中打开VR/AR应用,进入VR/AR应用场景,将手部伸入到深度摄像头的视场以内,即可在应用场景中出现对应数目的手的三维模型。 Opened from the smartphone VR / AR applications into VR / AR scenarios, projecting into the hand within the field of view of the depth camera, a corresponding number of three-dimensional model of the hand can appear in the application scenario. 通过手部姿态估计算法模拟真实世界中手的不同姿态,从而触发不同的手势交互动作。 Estimated pose different algorithms to simulate real-world hand by hand posture, gesture interaction to trigger different actions. 实现在VR/AR应用场景中的裸手凌空操作,从而提高VR/AR应用的真实感和沉浸感。 Implemented in VR / AR application scenarios bare hands volley operation, thereby enhancing the realism and immersion VR / AR applications.

[0060]本申请实施例提供的技术方案中,使用硬件包含一个VR/AR头戴式设备、智能移动设备和与之固定连接的一个深度传感器(该传感器采用结构光相机或TOF相机),利用深度传感器获取手部的深度点云数据,再通过手部姿态估计算法,能够精确估计出手部的骨骼自由度信息和关节点的三维位置信息,最终实现在虚拟现实和增强现实应用中的交互动作。 Technical solutions provided by the [0060] embodiment of the present application, the use of the hardware comprising a VR / AR headsets, smart mobile devices with a depth sensor and fixedly connected (the configuration using an optical sensor or a camera TOF camera) using depth sensor acquiring a depth point cloud data of the hand, then the hand posture estimation algorithm to accurately estimate the three-dimensional position information of freedom of the bone and joint hand portion of the information point, and ultimately in the virtual reality and augmented reality applications the interaction . 由于手部不需要增加额外的传感器设备,并且整个算法的运算都只依赖于手机和深度传感器的硬件单元,因此可以满足移动设备对算法效率、精度、便携式的要求。 Due to a hand no additional sensor device, and the entire operation is only dependent on the hardware algorithm phone unit and a depth sensor, and therefore meet the requirements of the mobile device algorithm efficiency, accuracy, portable.

[0061]本领域技术人员应当理解,本申请中所涉及的发明范围,并不限于上述技术特征的特定组合而成的技术方案,同时也应涵盖在不脱离所述发明构思的情况下,由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。 [0061] Those skilled in the art should understand that the scope of the present invention herein involved, not limited to the technical features of a particular combination of technical solution should also be covered without departing from the spirit of the invention comprises other technical features of the above aspect or any combination of equivalents formed. 例如上述特征与本申请中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。 The above features and, for example, disclosed in the present application (without limitation) having technical features similar functionality to replace another aspect formed.

Claims (9)

1.一种三维手势识别方法,其特征在于,所述方法包括: 获取用户手部的第一三维位置信息,所述第一三维位置信息是用户手部上第一位置点的位置信息; 使用预先定义的手势预测算法对所述第一三维位置信息进行预测计算,得到用户手部的第二三维位置信息和姿态信息,所述第二三维位置信息是用户手部上第二位置点的位置信息;以及将所述第二三维位置信息和所述姿态信息输入预先构建的三维手部模型,得到用户手部对应的三维手势。 1. A three-dimensional gesture recognition method, characterized in that, said method comprising: obtaining a first three-dimensional position information of the user's hand, the first three-dimensional position information is position information of the first user's hand position point; using predefined gestures prediction algorithm to the first three-dimensional position information prediction calculation to obtain a second dimensional position information and attitude information of the user's hand, a second three-dimensional position information of the user's hand is a position on a second position of the point information; and the three-dimensional model of the second three-dimensional hand position information and posture information input to the pre-built, to give the corresponding three-dimensional user's hand gestures.
2.根据权利要求1所述的方法,其特征在于,获取用户手部的第一三维位置信息包括: 通过深度传感器获取用户手部及周围环境的深度图像信息;以及根据预置的手部形状和深度特征值,从所述深度图像信息中提取出所述第一三维位置ί目息O And a hand unit in accordance with a preset shape; obtaining the user's hand and the depth image through the depth information of the surrounding environment sensor: 2. The method according to claim 1, wherein obtaining a first three-dimensional position information of the user's hand comprises and wherein the depth value, the first three-dimensional position of the extracted destination information ί O depth from the image information
3.根据权利要求2所述的方法,其特征在于,所述深度传感器为结构光相机、或飞行时间TOF相机。 3. The method according to claim 2, wherein said depth sensor is a structured light camera, or the time of flight TOF camera.
4.根据权利要求1所述的方法,其特征在于,所述手势预测算法是根据预先定义的深度学习算法对多个深度训练数据进行学习后得到的深度训练模型。 4. The method according to claim 1, wherein the depth of the gesture model prediction algorithm is trained to learn the depth of the plurality of training data according to a predefined depth learning algorithm obtained.
5.根据权利要求1至4中任一项所述的方法,其特征在于,所述第一位置点为用户手部的轮廓点,所述第二位置点为用户手部的关节点,所述姿态信息为用户手部的骨骼之间的角度。 The method according to any one of claim 1 to claim 4, wherein said first contour point location point of the user's hand, said second articular point position of the point of the user's hand, the between the bones of the user's hand posture angle information described later.
6.一种三维手势识别系统,包括穿戴式设备和终端设备,其特征在于: 所述穿戴式设备包括: 用于置于用户头部的头部固定结构; 深度传感器,用于获取用户手部及周围环境的深度图像信息; 接口传输结构,设置在所述深度传感器上,通过所述接口传输结构,所述深度传感器能够将所述深度图像信息发送给能够拆卸式安装在所述穿戴式设备上的终端设备; 所述终端设备包括: 提取模块,用于根据预置的手部形状和深度特征值,从所述深度图像信息中提取出所述第一三维位置信息; 计算模块,用于使用预先定义的手势预测算法对所述第一三维位置信息进行预测计算,得到用户手部的第二三维位置信息和姿态信息,所述第二三维位置信息是用户手部上第二位置点的位置信息; 处理模块,用于将所述第二三维位置信息和所述姿态信息输入预先构建的三维 A three-dimensional gesture recognition system, comprising a wearable device and a terminal device, wherein: said wearable device comprising: a fixing structure for a head-mounted head of a user; a depth sensor, for acquiring the user's hand the image and depth information of the surrounding environment; interface transfer structure disposed on the depth sensor, transmitted via the interface structure, the depth of the depth image sensor capable of transmitting information to be detachably mountable to the wearable device the terminal device; the terminal device comprising: an extracting module, according to the hand shape and depth preset characteristic value is extracted from the first three-dimensional position information of the depth image information; calculating module, for using predefined gesture prediction algorithm to the first three-dimensional position information prediction calculation to obtain a second dimensional position information and attitude information of the user's hand, a second three-dimensional position information on the user's hand portion is a second position of the point position information; processing module, for the second three-dimensional position information and posture information input to the pre-built three-dimensional 手部模型,得到用户手部对应的三维手势。 Hand model, the user's hand to give the corresponding three-dimensional gesture.
7.根据权利要求6所述的系统,其特征在于,所述深度传感器为结构光相机、或飞行时间TOF相机。 7. The system according to claim 6, wherein said depth sensor is a structured light camera, or the time of flight TOF camera.
8.根据权利要求6所述的系统,其特征在于,所述手势预测算法是根据预先定义的深度学习算法对多个深度训练数据进行学习后得到的深度训练模型。 8. The system according to claim 6, characterized in that the depth of the gesture prediction algorithm is trained model learning algorithm based on the depth obtained after a predefined depth of the plurality of training data for learning.
9.根据权利要求6至8中任一项所述的系统,其特征在于,所述第一位置点为用户手部的轮廓点,所述第二位置点为用户手部的关节点,所述姿态信息为用户手部的骨骼之间的角度。 9. The system as claimed in any one of claims 6-8, wherein said first contour point location point of the user's hand, said second articular point position of the point of the user's hand, the between the bones of the user's hand posture angle information described later.
CN201610694390.6A 2016-08-19 2016-08-19 Method and system for three-dimensional gesture recognition CN106325509A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610694390.6A CN106325509A (en) 2016-08-19 2016-08-19 Method and system for three-dimensional gesture recognition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610694390.6A CN106325509A (en) 2016-08-19 2016-08-19 Method and system for three-dimensional gesture recognition

Publications (1)

Publication Number Publication Date
CN106325509A true CN106325509A (en) 2017-01-11

Family

ID=57744382

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610694390.6A CN106325509A (en) 2016-08-19 2016-08-19 Method and system for three-dimensional gesture recognition

Country Status (1)

Country Link
CN (1) CN106325509A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011106797A1 (en) * 2010-02-28 2011-09-01 Osterhout Group, Inc. Projection triggering through an external marker in an augmented reality eyepiece
CN103839040A (en) * 2012-11-27 2014-06-04 株式会社理光 Gesture identification method and device based on depth images
CN104589356A (en) * 2014-11-27 2015-05-06 北京工业大学 Dexterous hand teleoperation control method based on Kinect human hand motion capturing
CN105138119A (en) * 2015-08-04 2015-12-09 湖南七迪视觉科技有限公司 Stereo vision system with automatic focusing controlled based on human biometrics
CN105302295A (en) * 2015-09-07 2016-02-03 哈尔滨市一舍科技有限公司 Virtual reality interaction device having 3D camera assembly
CN105759967A (en) * 2016-02-19 2016-07-13 电子科技大学 Global hand gesture detecting method based on depth data

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011106797A1 (en) * 2010-02-28 2011-09-01 Osterhout Group, Inc. Projection triggering through an external marker in an augmented reality eyepiece
CN103839040A (en) * 2012-11-27 2014-06-04 株式会社理光 Gesture identification method and device based on depth images
CN104589356A (en) * 2014-11-27 2015-05-06 北京工业大学 Dexterous hand teleoperation control method based on Kinect human hand motion capturing
CN105138119A (en) * 2015-08-04 2015-12-09 湖南七迪视觉科技有限公司 Stereo vision system with automatic focusing controlled based on human biometrics
CN105302295A (en) * 2015-09-07 2016-02-03 哈尔滨市一舍科技有限公司 Virtual reality interaction device having 3D camera assembly
CN105759967A (en) * 2016-02-19 2016-07-13 电子科技大学 Global hand gesture detecting method based on depth data

Similar Documents

Publication Publication Date Title
Ren et al. Depth camera based hand gesture recognition and its applications in human-computer-interaction
EP2904472B1 (en) Wearable sensor for tracking articulated body-parts
US10241639B2 (en) Dynamic user interactions for display control and manipulation of display objects
JP5887775B2 (en) Human Computer Interaction System, Hands designated point positioning method, and the gesture determination method of hand
Suarez et al. Hand gesture recognition with depth images: A review
Cruz et al. Kinect and rgbd images: Challenges and applications
Betancourt et al. The evolution of first person vision methods: A survey
Yeo et al. Hand tracking and gesture recognition system for human-computer interaction using low-cost hardware
US9945660B2 (en) Systems and methods of locating a control object appendage in three dimensional (3D) space
Wang et al. Real-time hand-tracking with a color glove
US20120306850A1 (en) Distributed asynchronous localization and mapping for augmented reality
CN101673161B (en) A visual entity operable without a touch screen system
US9696795B2 (en) Systems and methods of creating a realistic grab experience in virtual reality/augmented reality environments
CN102665838B (en) Methods and systems for determining and tracking extremities of a target
US9372544B2 (en) Gesture recognition techniques
Qian et al. Developing a gesture based remote human-robot interaction system using Kinect
US8269722B2 (en) Gesture recognition system and method thereof
CN203224887U (en) Display control device
US20130010071A1 (en) Methods and systems for mapping pointing device on depth map
CN102184020A (en) Method for manipulating posture of user interface and posture correction
CN102915111A (en) Wrist gesture control system and method
JPH10320588A (en) Picture processor and picture processing method
CN103366610A (en) Augmented-reality-based three-dimensional interactive learning system and method
CN102789313B (en) User interaction system and method
WO2014022448A1 (en) Animating objects using the human body

Legal Events

Date Code Title Description
PB01
C10 Entry into substantive examination