WO2018076484A1 - 一种基于视频的捏合指尖跟踪方法 - Google Patents

一种基于视频的捏合指尖跟踪方法 Download PDF

Info

Publication number
WO2018076484A1
WO2018076484A1 PCT/CN2016/109614 CN2016109614W WO2018076484A1 WO 2018076484 A1 WO2018076484 A1 WO 2018076484A1 CN 2016109614 W CN2016109614 W CN 2016109614W WO 2018076484 A1 WO2018076484 A1 WO 2018076484A1
Authority
WO
WIPO (PCT)
Prior art keywords
fingertip
pinch
video frame
region
skin color
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/109614
Other languages
English (en)
French (fr)
Inventor
康文雄
吴桂乐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Priority to SG11201900381PA priority Critical patent/SG11201900381PA/en
Publication of WO2018076484A1 publication Critical patent/WO2018076484A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/56—Extraction of image or video features relating to colour
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/48—Matching video sequences
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/49—Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20048—Transform domain processing
    • G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the present invention relates to the field of image processing and analysis technologies, and more particularly to a video-based pinch fingertip tracking method.
  • the situation of human hand writing is simulated mainly by extracting the trajectory of a single fingertip before the camera.
  • this way of writing in the air does not really simulate the actual situation of human hand writing - because when a person writes with a pen, it usually uses three fingers to pinch the pen instead of only a single finger. Therefore, the use of a single finger to simulate the handwriting method not only fails to correctly reflect the habits of hand writing, but also greatly limits the flexibility of hand writing, resulting in a large difference between the air writing track and the actual handwriting.
  • the kneading fingertips in the air writing obviously more closely simulates the actual posture when the pen is written by the human hand, and more realistically reflects the actual handwriting.
  • the tracking and recognition method of the pinch fingertip is still blank, and the shape of the pinch fingertip is complicated and the tracking is difficult. Therefore, it is necessary to design a kneading fingertip tracking method, so that the kneading fingertip writing mode can be applied to the aerial writing system. in.
  • the object of the present invention is to overcome the shortcomings and deficiencies in the prior art, and to provide a video-based pinch fingertip tracking method, which can extract effective skin color and texture features to achieve effective tracking recognition of the pinch fingertip.
  • a video based A pinch fingertip tracking method for continuously tracking a pinch finger in air writing characterized in that it comprises the following five steps:
  • the video frame image is captured, the video frame image is converted from the RGB image to the YCbCr image, the skin color region is segmented by the YCbCr skin color detector, and the skin region is segmented by the fist detector to obtain the fist region, which is used in the fist region. Kneading the fingertip cascade classifier to position and pinch the fingertip area to determine the initial position of the pinch fingertip;
  • the second step is to capture the next frame of the video frame as the current processed video frame map; the center of the pinch fingertip region obtained from the previous frame of the video frame is centered on the current processed video frame map to establish a target search region; using YCbCr skin color detection Extracting the skin color region of the target search area to obtain the skin color target search region of the current processed video frame image; using the Canny edge detector to respectively search the skin color target search region of the previous frame of the video frame image and the skin color target image of the currently processed video frame image
  • the area is processed to obtain the contour, and the contour point is obtained by using the dense sampling method; the contour point of the previous frame video frame picture and the currently processed video frame picture is matched by the LK optical flow method to determine the motion distance of the skin color target search area;
  • the third step is to construct a pinch fingertip skin color mapping table; divide the pinch fingertip area of the previous frame of the video frame into sixteen area units, and respectively find each area unit CbCr color band is closest in the pinch fingertip skin color mapping table. The value is combined with the gray value of each regional unit to obtain the CbCr-Gray color feature; the previous frame of the video frame is calculated to knead the HOG feature of the fingertip region, thereby obtaining the CbCr-Gray-HOG feature of the pinched fingertip region;
  • a filter tracking method based on a Gaussian kernel function is used, and the CbCr-Gray-HOG feature of the pinch fingertip region is combined to determine a center point of the pinch fingertip region of the currently processed video frame image;
  • the fifth step it is judged whether the kneading fingertip tracking end condition is established: if it is established, the tracking ends; otherwise, it jumps to the second step.
  • the video frame image is captured, the video frame image is converted from the RGB image to the YCbCr image, the skin color region is segmented by the YCbCr skin color detector, and the skin region is segmented by the fist detector to obtain the fist region.
  • the kneading fingertip cascade classifier is used to position and pinch the fingertip area, and determining the initial position of the pinch fingertip refers to: including the following steps:
  • step (1.3) constructing the YCbCr skin color filter means separating the YCbCr image into the Y channel, the Cb channel and the Cr channel, and selecting the Cb channel value in the ribbon range
  • the area within the [77, 127] and the Cr channel value in the ribbon range [133, 173] is used as the skin color area.
  • the threshold value T1 is set to one tenth of the width of the pinched fingertip region; the threshold value T2 is set to one tenth of the height of the pinched fingertip region.
  • the capturing the next frame of the video frame is used as the current processed video frame map; the center of the pinch fingertip region obtained by the video frame image of the previous frame is centered on the current processed video frame map to establish a target search region;
  • the YCbCr skin color detector is used to extract the skin color region of the target search area, thereby obtaining the skin color target search region of the current processed video frame image; and the Canny edge detector is used for the skin color target search region of the previous frame video frame image and the current processed video frame map.
  • the skin color target search area is processed to obtain contours, and the contour points are acquired by using dense sampling method; the contour points of the previous frame video frame map and the currently processed video frame map are matched by the LK optical flow method to determine the motion of the skin color target search region.
  • Distance means including the following steps:
  • the Canny edge detector is used to process the skin color target search area of the previous frame of the video frame and the skin color target search area of the currently processed video frame picture to obtain the contour, and adopt dense sampling. Method for obtaining contour points;
  • the kneading fingertip skin color mapping table is constructed; the pinch fingertip region of the previous frame of the video frame is divided into sixteen region units, and each region unit CbCr ribbon is searched for the pinch fingertip skin color mapping.
  • the closest value in the table combined with the gray value of each regional unit, obtains the CbCr-Gray color feature; calculates the HOG feature of the previous frame of the video frame to pinch the fingertip region, thereby obtaining the CbCr-Gray-HOG feature of the pinched fingertip region.
  • the CbCr-Gray color feature of the sixteen-dimensional kneading fingertip region is combined with the HOG feature of the thirty-one kneading fingertip region to obtain the CbCr-Gray-HOG feature of the forty-seven-dimensional kneading fingertip region.
  • the Gaussian kernel function-based filter tracking method is combined with the CbCr-Gray-HOG feature of the pinch fingertip region to determine the center point of the pinch fingertip region of the currently processed video frame map: The following steps:
  • ⁇ is the Hilbert space mapping function
  • ⁇ is the regularization parameter
  • w is the closed form solution
  • ⁇ is the coefficient
  • F and F -1 are the discrete Fourier transform and the discrete Fourier inverse transform, respectively.
  • * is a conjugate complex number
  • p is an image block matrix represented by a CbCr-Gray-HOG feature of the pinch fingertip region
  • p' is a transpose of the feature image block matrix p
  • p c is a feature The feature image block of the c-th dimension in the image block matrix p
  • ⁇ 2 is the variance in the Gaussian function
  • ⁇ is the multiplication between the elements of the Fourier domain
  • the determining whether the pinch fingertip tracking end condition is established means that the threshold values T3 and T4 are set, and determining the pinch fingertip region of the adjacent two frames of the video frame image in the consecutive video frame frames of several frames Whether the abscissa change value and the ordinate change value of the center point are smaller than T3 and T4, respectively: If yes, the pinch fingertip tracking end condition is established, otherwise it is not established.
  • the threshold T3 is set to a tenth of the width of the pinched fingertip region; the threshold T4 is set to one tenth of the height of the pinched fingertip region.
  • the present invention has the following advantages and benefits:
  • the kneading fingertip tracking method of the present invention is applied to an aerial writing system to accurately identify a pinch fingertip
  • the trajectory of motion makes the kneading fingertip air writing system realize; unlike the air writing method of a single fingertip, the pinch fingertip air writing more closely simulates the actual posture when the pen is written by the human hand, and retains the flexibility of the handwriting trajectory. , making the video-based aerial writing system truly reflect the actual handwriting situation;
  • the kneading fingertip tracking method of the present invention comprises a YCbCr skin color detector, a haar-like feature based fist detector and a HOG feature based pinch fingertip cascade classifier in the initialization phase to form a multi-component correlation detector, from complex Accurately locate the position of the pinch fingertip; then, extract the skin color target search area through the YCbCr skin color detector, use the Canny edge detector to acquire the contour and sample the contour point, and then use the LK optical flow method to perform front and back frame matching of the contour point.
  • the pinch fingertip skin color mapping table mapping the pinching fingertip area to the pinch fingertip skin color mapping table, combining the gray value to obtain the CbCr-Gray color feature, and calculating the pinch finger
  • the HOG feature of the sharp region so as to obtain the CbCr-Gray-HOG feature of the pinched fingertip region
  • the Gaussian kernel function-based filter tracking method is used to locate and determine the currently processed video frame.
  • the pinch of the map is centered on the fingertip area and can be effectively tracked by extracting effective skin tones and texture features.
  • 1 is a schematic diagram showing the simulation of the actual handwriting situation of the pinch fingertip tracking method of the present invention
  • FIG. 2 is a flow chart of a kneading fingertip tracking method of the present invention and its application in an air signature identity authentication system;
  • FIG. 3 is a flow chart showing a method for detecting the initial position of the pinch fingertip in the first initialization stage of the kneading fingertip tracking method of the present invention
  • FIG. 4 is a schematic diagram of a process of constructing a skin color mapping table and extracting multidimensional features in the third step of the kneading fingertip tracking method of the present invention.
  • a video-based pinch fingertip tracking method is used for continuously tracking a pinch finger in an air writing; the following five steps are included:
  • the video frame image is captured, the video frame image is converted from the RGB image to the YCbCr image, the skin color region is segmented by the YCbCr skin color detector, and the skin region is segmented by the fist detector to obtain the fist region, which is used in the fist region. Kneading the fingertip cascade classifier to position and pinch the fingertip area to determine the initial position of the pinch fingertip;
  • the second step is to capture the next frame of the video frame as the current processed video frame map; the center of the pinch fingertip region obtained from the previous frame of the video frame is centered on the current processed video frame map to establish a target search region; using YCbCr skin color detection Extracting the skin color region of the target search area to obtain the skin color target search region of the current processed video frame image; using the Canny edge detector to respectively search the skin color target search region of the previous frame of the video frame image and the skin color target image of the currently processed video frame image
  • the area is processed to obtain the contour, and the contour point is obtained by using the dense sampling method; the contour point of the previous frame video frame picture and the currently processed video frame picture is matched by the LK optical flow method to determine the motion distance of the skin color target search area;
  • the third step is to construct a pinch fingertip skin color mapping table; divide the pinch fingertip area of the previous frame of the video frame into sixteen area units, and respectively find each area unit CbCr color band is closest in the pinch fingertip skin color mapping table. The value is combined with the gray value of each regional unit to obtain the CbCr-Gray color feature; the previous frame of the video frame is calculated to knead the HOG feature of the fingertip region, thereby obtaining the CbCr-Gray-HOG feature of the pinched fingertip region;
  • a filter tracking method based on a Gaussian kernel function is used, and the CbCr-Gray-HOG feature of the pinch fingertip region is combined to determine a center point of the pinch fingertip region of the currently processed video frame image;
  • the fifth step it is judged whether the kneading fingertip tracking end condition is established: if it is established, the tracking ends; otherwise, it jumps to the second step.
  • the video frame image is captured, the video frame image is converted from the RGB image to the YCbCr image, the skin color region is segmented by the YCbCr skin color detector, and the skin region is segmented by the fist detector to obtain the fist region.
  • the kneading fingertip cascade classifier is used to position and pinch the fingertip area, and determining the initial position of the pinch fingertip refers to: including the following steps:
  • the YCbCr skin color filter is constructed; the skin color area in each YCbCr image is determined by the YCbCr skin color filter, and the non-skin color area is eliminated to eliminate the interference of the illumination and the non-skin area;
  • the first step is the initialization phase, and the kneading fingertip tracking method of the present invention keeps the pinch fingertips at the same position for a certain period of time as a judgment condition for the start of the air writing tracking; since the writer's hand may have slight jitter and image recognition may There is an error, and therefore it is judged whether or not the kneading fingertip is at the same position by judging whether the horizontal and vertical coordinate change value of the center point of the fingertip region is within a certain range.
  • step (1.3) constructing the YCbCr skin color filter means separating the YCbCr image into the Y channel, the Cb channel and the Cr channel, and selecting the Cb channel value in the ribbon range
  • the area within the [77, 127] and the Cr channel value in the ribbon range [133, 173] is used as the skin color area.
  • step (1.4) offline training based on the Haar-like feature of the fist classifier means that the image including the fist is used as the positive sample, the image not including the fist is used as the negative sample, and the Haar-like feature is used to align the positive sample and the negative sample.
  • Learning, generating a fist classifier, the target of the fist classifier is a fist.
  • step (1.5) the offline training of the kneading fingertip cascade classifier based on the HOG feature means that an image including a pinch fingertip is used as a positive sample, an image not including a pinch fingertip is used as a negative sample, and the HOG feature is aligned.
  • the sample and the negative sample are learned to generate a pinch fingertip cascade classifier, and the pinch fingertip cascade classifier recognizes the target by pinching the fingertip.
  • the threshold value T1 is set to one tenth of the width of the pinched fingertip region; the threshold value T2 is set to one tenth of the height of the pinched fingertip region.
  • the capturing the next frame of the video frame is used as the current processed video frame map; the center of the pinch fingertip region obtained by the video frame image of the previous frame is centered on the current processed video frame map to establish a target search region;
  • the YCbCr skin color detector is used to extract the skin color region of the target search area, thereby obtaining the skin color target search region of the current processed video frame image; and the Canny edge detector is used for the skin color target search region of the previous frame video frame image and the current processed video frame map.
  • Skin color target search area for processing The contour is obtained, and the contour points are obtained by using the dense sampling method; the contour points of the previous frame video frame map and the currently processed video frame map are matched by the LK optical flow method, and the motion distance of the skin color target search area is determined to be: :
  • the Canny edge detector is used to process the skin color target search area of the previous frame of the video frame and the skin color target search area of the currently processed video frame picture to obtain the contour, and obtain the contour point by using the dense sampling method;
  • the kneading fingertip skin color mapping table is constructed; the pinch fingertip region of the previous frame of the video frame is divided into sixteen region units, and each region unit CbCr ribbon is searched for the pinch fingertip skin color mapping.
  • the closest value in the table combined with the gray value of each regional unit, obtains the CbCr-Gray color feature; calculates the HOG feature of the previous frame of the video frame to pinch the fingertip region, thereby obtaining the CbCr-Gray-HOG feature of the pinched fingertip region.
  • the CbCr-Gray color feature of the sixteen-dimensional kneading fingertip region is combined with the HOG feature of the thirty-one kneading fingertip region to obtain the CbCr-Gray-HOG feature of the forty-seven-dimensional kneading fingertip region.
  • selecting the skin color of the sixteen different color bands to form a pinch fingertip skin color mapping table refers to selecting four basic skin color values for ivory, linen, lemon yellow, and brown, and in each skin color value. Three skin color values are selected nearby to form a skin color of a total of sixteen different color bands to form a pinch fingertip skin color mapping table.
  • the Gaussian kernel function-based filter tracking method is combined with the CbCr-Gray-HOG feature of the pinch fingertip region to determine the center point of the pinch fingertip region of the currently processed video frame map: The following steps:
  • ⁇ is the Hilbert space mapping function
  • ⁇ is the regularization parameter
  • w is the closed form solution
  • ⁇ is the coefficient
  • F and F -1 are the discrete Fourier transform and the discrete Fourier inverse transform, respectively.
  • * is a conjugate complex number
  • p is an image block matrix represented by a CbCr-Gray-HOG feature of the pinch fingertip region
  • p' is a transpose of the feature image block matrix p
  • p c is a feature The feature image block of the c-th dimension in the image block matrix p
  • ⁇ 2 is the variance in the Gaussian function
  • ⁇ is the multiplication between the elements of the Fourier domain
  • the determining whether the pinch fingertip tracking end condition is established means that the threshold values T3 and T4 are set, and determining the pinch fingertip region of the adjacent two frames of the video frame image in the consecutive video frame frames of several frames Whether the abscissa change value and the ordinate change value of the center point are smaller than T3 and T4, respectively: If yes, the pinch fingertip tracking end condition is established, otherwise it is not established.
  • the kneading fingertip tracking method of the present invention keeps the pinch fingertips in a small area of the same position for a certain period of time as a criterion for judging the end of the tracking.
  • the threshold T3 is set to a tenth of the width of the pinched fingertip region; the threshold T4 is set to one tenth of the height of the pinched fingertip region.
  • the kneading fingertip tracking method of the invention is applied to an aerial writing system, and can accurately recognize the kinematic movement of the pinch fingertip, thereby realizing the kneading fingertip air writing system; unlike the air writing mode of a single fingertip, the kneading fingertip air writing is more It aptly simulates the actual posture when the pen is written by the human hand, retaining the flexibility of the handwriting trajectory, so that the video-based aerial writing system truly reflects the actual handwriting situation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

本发明提供了一种基于视频的捏合指尖跟踪方法，其特征在于：首先采用YCbCr肤色检测器、拳头检测器和捏合指尖级联分类器构成多成分相关检测器，定位捏合指尖的位置；然后提取肤色目标搜索区域，获取轮廓并采样轮廓点，通过前后帧匹配确定肤色目标搜索区域的运动距离；之后将捏合指尖区域映射到捏合指尖肤色映射表中，结合灰度值得到CbCr-Gray颜色特征，并计算HOG特征，从而得到CbCr-Gray-HOG特征；最后，采用基于高斯核函数的滤波跟踪方法定位确定当前处理视频帧图的捏合指尖区域中心点。该方法应用于空中书写系统中，可准确识别捏合指尖运动轨迹，从而使捏合指尖空中书写系统得以实现。

Description

一种基于视频的捏合指尖跟踪方法 技术领域
本发明涉及图像处理与分析技术领域，更具体地说，涉及一种基于视频的捏合指尖跟踪方法。
背景技术
近年来，随着虚拟现实和增强现实的快速发展，人机交互技术得到极大的发展，吸引越来越多的研究人员关注。当前，作为人机交互的核心技术之一，手指检测与跟踪被广泛应用于各个领域，诸如手势识别、空中书写等。其中，基于视频的空中书写已经被广泛应用于空中书写系统的字符识别、轨迹认证等领域。
在传统的空中书写系统中，主要采用提取摄像头前单个指尖运动轨迹的方式模拟人手书写的情况。然而，这种空中书写方式并不能真正模拟人手书写的实际情况——因为人手拿笔书写时，采用的一般都是三个手指捏合笔的方式，而不是只有单个手指。因此，采用单个手指模拟人手书写的方式，不仅无法正确反映人手书写时的习惯，而且极大地限制了人手书写的灵活性，导致空中书写轨迹与实际笔迹存在较大差异。
与单个手指空中书写相比，捏合指尖空中书写显然更贴切地模拟了人手拿笔书写时的实际姿态，更真实地反映实际手写情况。但是目前捏合指尖的跟踪识别方法仍处于空白，且捏合指尖形态较为复杂，跟踪难度较大；因此需要设计出一种捏合指尖跟踪方法，使捏合指尖书写方式能应用到空中书写系统中。
发明内容
本发明的目的在于克服现有技术中的缺点与不足，提供一种基于视频的捏合指尖跟踪方法，该捏合指尖跟踪方法可提取有效的肤色和纹理特征实现捏合指尖的有效跟踪识别。
为了达到上述目的，本发明通过下述技术方案予以实现：一种基于视频的 捏合指尖跟踪方法，用于对空中书写中的捏合手指进行持续跟踪；其特征在于：包括以下五个步骤：
第一步，捕获视频帧图，将视频帧图从RGB图像转化为YCbCr图像，通过YCbCr肤色检测器分割出肤色区域，采用拳头检测器对肤色区域进行分割得出拳头区域，在拳头区域中采用捏合指尖级联分类器定位捏合指尖区域，确定捏合指尖初始位置；
第二步，捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样方法获取轮廓点；采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，确定肤色目标搜索区域的运动距离；
第三步，构建捏合指尖肤色映射表；将上一帧视频帧图的捏合指尖区域划分为十六个区域单元，分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，结合各个区域单元的灰度值得到CbCr-Gray颜色特征；计算上一帧视频帧图捏合指尖区域的HOG特征，从而得到捏合指尖区域的CbCr-Gray-HOG特征；
第四步，采用基于高斯核函数的滤波跟踪方法，结合所述捏合指尖区域的CbCr-Gray-HOG特征，确定当前处理视频帧图的捏合指尖区域中心点；
第五步，判断捏合指尖跟踪结束条件是否成立：若成立，则跟踪结束；否则跳至第二步。
在第一步中，所述捕获视频帧图，将视频帧图从RGB图像转化为YCbCr图像，通过YCbCr肤色检测器分割出肤色区域，采用拳头检测器对肤色区域进行分割得出拳头区域，在拳头区域中采用捏合指尖级联分类器定位捏合指尖区域，确定捏合指尖初始位置是指：包括以下步骤：
(1.1)捕获若干帧连续的视频帧图；
(1.2)将各帧视频帧图像分别从RGB图像转化为YCbCr图像；
(1.3)根据人体肤色聚类特性，构造YCbCr肤色滤波器；采用YCbCr肤色滤波器确定各个YCbCr图像中的肤色区域，将非肤色区域剔除从而消除光照 和非肤色区域的干扰；
(1.4)离线训练基于Haar-like特征的拳头分类器，并用拳头分类器定位各个肤色区域中的拳头区域；
(1.5)离线训练基于HOG特征的捏合指尖级联分类器，并用捏合指尖级联分类器定位各个拳头区域中的捏合指尖区域；
(1.6)设定阈值T1和T2；判断各个捏合指尖区域中心点的横坐标变化最大值和纵坐标变化最大值：如果横坐标变化最大值≤T1，且纵坐标变化最大值≤T2，则设定所述若干帧连续的视频帧图中最后一帧视频帧图的捏合指尖区域中心点为捏合指尖初始位置，并跳至第二步；否则捕获后续若干帧连续的视频帧图，并跳至步骤(1.2)。
优选地，在步骤(1.3)中，所述根据人体肤色聚类特性，构造YCbCr肤色滤波器是指，将YCbCr图像进行Y通道、Cb通道和Cr通道分离，选定Cb通道数值在色带范围[77,127]内且Cr通道数值在色带范围[133,173]内的区域作为肤色区域。
在步骤(1.6)中，所述阈值T1设定为捏合指尖区域宽度的十分之一；所述阈值T2设定为捏合指尖区域高度的十分之一。
在第二步中，所述捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样方法获取轮廓点；采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，确定肤色目标搜索区域的运动距离是指：包括以下步骤：
(2.1)捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；
(2.2)采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样 方法获取轮廓点；
(2.3)采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，对上一帧视频帧图和当前处理视频帧图中各个对应轮廓点分别计算出运动距离，将运动距离的均值或中间值定义为肤色目标搜索区域的运动距离。
在第三步中，所述构建捏合指尖肤色映射表；将上一帧视频帧图的捏合指尖区域划分为十六个区域单元，分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，结合各个区域单元的灰度值得到CbCr-Gray颜色特征；计算上一帧视频帧图捏合指尖区域的HOG特征，从而得到捏合指尖区域的CbCr-Gray-HOG特征是指：包括以下步骤：
(3.1)基于人体肤色的聚类特性，选择十六种不同色带的肤色构成捏合指尖肤色映射表；
(3.2)将上一帧视频帧图的捏合指尖区域划分为十六个区域单元并映射到肤色映射表中，以分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，分别计算各个区域单元的灰度值，得到十六维捏合指尖区域的CbCr-Gray颜色特征；
(3.3)计算上一帧视频帧图捏合指尖区域在九个梯度方向上的HOG特征，每个方向分别包括三维，共得到二十七维HOG特征，结合五维基础HOG特征，并剔除梯度值为全零的一维冗余HOG特征，构成三十一维捏合指尖区域的HOG特征；
(3.4)十六维捏合指尖区域的CbCr-Gray颜色特征与三十一维捏合指尖区域的HOG特征结合得到四十七维捏合指尖区域的CbCr-Gray-HOG特征。
在第四步中，所述采用基于高斯核函数的滤波跟踪方法，结合所述捏合指尖区域的CbCr-Gray-HOG特征，确定当前处理视频帧图的捏合指尖区域中心点是指：包括以下步骤：
(4.1)将上一帧视频帧图的捏合指尖区域中心点偏移所述肤色目标搜索区域的运动距离后得到的点作为中心在当前处理视频帧图上划定采样区域；在采样区域中，以上一帧视频帧图的捏合指尖区域中心点为中心建立大小为M×N的捏合指尖搜索区域；用循环矩阵对捏合指尖搜索区域进行循环采样得到若干捏合指尖候选样本pm,n，其中m、n分别是指捏合指尖候选样本pm,n的横、纵坐标值，m∈{0,...,M-1}，n∈{0,...,N-1}；
(4.2)用高斯函数分别标记各个捏合指尖候选样本pm,n；并采用公式(1)(2)、(3)和(4)计算各个捏合指尖候选样本pm,n的优化损失函数：
Figure PCTCN2016109614-appb-000001
w＝∑m,nα(m,n)φ(pm,n)              (2)
Figure PCTCN2016109614-appb-000002
Figure PCTCN2016109614-appb-000003
其中，φ是希尔伯特空间映射函数，λ是正则化参数，w是闭合形式的解，α是系数，F和F-1分别为离散傅里叶变换和离散傅里叶逆变换，y是回归函数标记图像块矩阵，*是共轭复数，p是用捏合指尖区域的CbCr-Gray-HOG特征表示的图像块矩阵，p'是特征图像块矩阵p的转置，pc是特征图像块矩阵p中第c维的特征图像块，σ2是高斯函数中的方差，⊙是傅里叶域的元素间相乘；
寻找公式(1)计算所得的最小值以获取损失最小值，得到损失最小值对应的系数α，将该系数α代入到公式(5)中：
Figure PCTCN2016109614-appb-000004
其中，z是捏合指尖搜索区域图像块，
Figure PCTCN2016109614-appb-000005
为核相关卷积后的响应值；
将各个捏合指尖候选样本pm,n分别代入到公式(5)中计算得到各个捏合指尖候选样本pm,n对应的
Figure PCTCN2016109614-appb-000006
将
Figure PCTCN2016109614-appb-000007
最小值对应的捏合指尖候选样本pm,n作为当前处理视频帧图的捏合指尖区域，捏合指尖候选样本pm,n对应的点(m,n)为当前处理视频帧图的捏合指尖区域中心点。
在第五步中，所述判断捏合指尖跟踪结束条件是否成立是指，设定阈值T3和T4，判断在若干帧连续的视频帧图中，相邻两帧视频帧图的捏合指尖区域中心点的横坐标变化值和纵坐标变化值是否分别小于T3和T4：若是，则捏合指尖跟踪结束条件成立，否则不成立。
所述阈值T3设定为捏合指尖区域宽度的十分之一；所述阈值T4设定为捏合指尖区域高度的十分之一。
与现有技术相比，本发明具有如下优点与有益效果：
1、本发明捏合指尖跟踪方法应用于空中书写系统中，可准确识别捏合指尖 运动轨迹，从而使捏合指尖空中书写系统得以实现；不同于单个指尖的空中书写方式，捏合指尖空中书写较为贴切地模拟了人手拿笔书写时的实际姿态，保留了手写轨迹的灵活性，使得基于视频的空中书写系统真实地反映实际的手写情况；
2、本发明捏合指尖跟踪方法，在初始化阶段采用YCbCr肤色检测器、基于Haar-like特征的拳头检测器和基于HOG特征的捏合指尖级联分类器构成一个多成分相关检测器，从复杂的场景中准确定位捏合指尖的位置；然后，通过YCbCr肤色检测器提取肤色目标搜索区域，用Canny边缘检测器获取轮廓并采样轮廓点，再用LK光流法进行轮廓点的前后帧匹配，从而确定肤色目标搜索区域的运动距离；之后，构建捏合指尖肤色映射表，将捏合指尖区域映射到捏合指尖肤色映射表中，结合灰度值得到CbCr-Gray颜色特征，并计算捏合指尖区域的HOG特征，从而得到捏合指尖区域的CbCr-Gray-HOG特征；最后，基于捏合指尖区域的CbCr-Gray-HOG特征，采用基于高斯核函数的滤波跟踪方法定位确定当前处理视频帧图的捏合指尖区域中心点，可通过提取有效的肤色和纹理特征并实现有效的跟踪。
附图说明
图1是本发明捏合指尖跟踪方法对实际手写情况的模拟示意图；
图2是本发明捏合指尖跟踪方法及其在空中签名身份认证系统中应用的流程框图；
图3是本发明捏合指尖跟踪方法中第一步初始化阶段检测捏合指尖初始位置的方法流程图；
图4是本发明捏合指尖跟踪方法中第三步构建肤色映射表并提取多维特征的过程示意图。
具体实施方式
下面结合附图与具体实施方式对本发明作进一步详细的描述。
实施例
如图1至图4所示，本发明一种基于视频的捏合指尖跟踪方法，用于对空中书写中的捏合手指进行持续跟踪；包括以下五个步骤：
第一步，捕获视频帧图，将视频帧图从RGB图像转化为YCbCr图像，通过YCbCr肤色检测器分割出肤色区域，采用拳头检测器对肤色区域进行分割得出拳头区域，在拳头区域中采用捏合指尖级联分类器定位捏合指尖区域，确定捏合指尖初始位置；
第二步，捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样方法获取轮廓点；采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，确定肤色目标搜索区域的运动距离；
第三步，构建捏合指尖肤色映射表；将上一帧视频帧图的捏合指尖区域划分为十六个区域单元，分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，结合各个区域单元的灰度值得到CbCr-Gray颜色特征；计算上一帧视频帧图捏合指尖区域的HOG特征，从而得到捏合指尖区域的CbCr-Gray-HOG特征；
第四步，采用基于高斯核函数的滤波跟踪方法，结合所述捏合指尖区域的CbCr-Gray-HOG特征，确定当前处理视频帧图的捏合指尖区域中心点；
第五步，判断捏合指尖跟踪结束条件是否成立：若成立，则跟踪结束；否则跳至第二步。
在第一步中，所述捕获视频帧图，将视频帧图从RGB图像转化为YCbCr图像，通过YCbCr肤色检测器分割出肤色区域，采用拳头检测器对肤色区域进行分割得出拳头区域，在拳头区域中采用捏合指尖级联分类器定位捏合指尖区域，确定捏合指尖初始位置是指：包括以下步骤：
(1.1)捕获若干帧连续的视频帧图；帧数由设计者设定；
(1.2)将各帧视频帧图像分别从RGB图像转化为YCbCr图像；
(1.3)根据人体肤色聚类特性，构造YCbCr肤色滤波器；采用YCbCr肤色滤波器确定各个YCbCr图像中的肤色区域，将非肤色区域剔除从而消除光照和非肤色区域的干扰；
(1.4)离线训练基于Haar-like特征的拳头分类器，并用拳头分类器定位各 个肤色区域中的拳头区域；
(1.5)离线训练基于HOG特征的捏合指尖级联分类器，并用捏合指尖级联分类器定位各个拳头区域中的捏合指尖区域；
(1.6)设定阈值T1和T2；判断各个捏合指尖区域中心点的横坐标变化最大值和纵坐标变化最大值：如果横坐标变化最大值≤T1，且纵坐标变化最大值≤T2，则设定所述若干帧连续的视频帧图中最后一帧视频帧图的捏合指尖区域中心点为捏合指尖初始位置，并跳至第二步；否则捕获后续若干帧连续的视频帧图，并跳至步骤(1.2)。
第一步为初始化阶段，本发明捏合指尖跟踪方法将捏合指尖是否在同一位置持续停留一定时间作为转入空中书写跟踪开始的判断条件；由于书写者手部可能存在轻微抖动以及图像识别可能存在误差，因此将捏合指尖区域中心点的横纵坐标变化值是否在一定范围内来判断捏合指尖是否在同一位置上。
优选地，在步骤(1.3)中，所述根据人体肤色聚类特性，构造YCbCr肤色滤波器是指，将YCbCr图像进行Y通道、Cb通道和Cr通道分离，选定Cb通道数值在色带范围[77,127]内且Cr通道数值在色带范围[133,173]内的区域作为肤色区域。
在步骤(1.4)中，离线训练基于Haar-like特征的拳头分类器是指，采用包括拳头的图像作为正样本，不包括拳头的图像作为负样本，采用Haar-like特征对正样本和负样本进行学习，生成拳头分类器，拳头分类器的识别目标是拳头。
在步骤(1.5)中，离线训练基于HOG特征的捏合指尖级联分类器是指，采用包括捏合指尖的图像作为正样本，不包括捏合指尖的图像作为负样本，采用HOG特征对正样本和负样本进行学习，生成捏合指尖级联分类器，捏合指尖级联分类器的识别目标是捏合指尖。
在步骤(1.6)中，所述阈值T1设定为捏合指尖区域宽度的十分之一；所述阈值T2设定为捏合指尖区域高度的十分之一。
在第二步中，所述捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获 取轮廓，并采用密集采样方法获取轮廓点；采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，确定肤色目标搜索区域的运动距离是指：包括以下步骤：
(2.1)捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；
(2.2)采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样方法获取轮廓点；
(2.3)采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，对上一帧视频帧图和当前处理视频帧图中各个对应轮廓点分别计算出运动距离，将运动距离的均值或中间值定义为肤色目标搜索区域的运动距离。
在第三步中，所述构建捏合指尖肤色映射表；将上一帧视频帧图的捏合指尖区域划分为十六个区域单元，分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，结合各个区域单元的灰度值得到CbCr-Gray颜色特征；计算上一帧视频帧图捏合指尖区域的HOG特征，从而得到捏合指尖区域的CbCr-Gray-HOG特征是指：包括以下步骤：
(3.1)基于人体肤色的聚类特性，选择十六种不同色带的肤色构成捏合指尖肤色映射表；
(3.2)将上一帧视频帧图的捏合指尖区域划分为十六个区域单元并映射到肤色映射表中，以分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，分别计算各个区域单元的灰度值，得到十六维捏合指尖区域的CbCr-Gray颜色特征；
(3.3)计算上一帧视频帧图捏合指尖区域在九个梯度方向上的HOG特征，每个方向分别包括三维，共得到二十七维HOG特征，结合五维基础HOG特征，并剔除梯度值为全零的一维冗余HOG特征，构成三十一维捏合指尖区域的HOG特征；
(3.4)十六维捏合指尖区域的CbCr-Gray颜色特征与三十一维捏合指尖区域的HOG特征结合得到四十七维捏合指尖区域的CbCr-Gray-HOG特征。
在步骤(3.1)中，选择十六种不同色带的肤色构成捏合指尖肤色映射表是指，选择象牙白、亚麻色、柠檬黄、棕色为四种基本肤色值，并在每种肤色数值附近再选择三种肤色值，组成一共十六种不同色带的肤色来构成捏合指尖肤色映射表。
在第四步中，所述采用基于高斯核函数的滤波跟踪方法，结合所述捏合指尖区域的CbCr-Gray-HOG特征，确定当前处理视频帧图的捏合指尖区域中心点是指：包括以下步骤：
(4.1)将上一帧视频帧图的捏合指尖区域中心点偏移所述肤色目标搜索区域的运动距离后得到的点作为中心在当前处理视频帧图上划定采样区域；在采样区域中，以上一帧视频帧图的捏合指尖区域中心点为中心建立大小为M×N的捏合指尖搜索区域；用循环矩阵对捏合指尖搜索区域进行循环采样得到若干捏合指尖候选样本pm,n，其中m、n分别是指捏合指尖候选样本pm,n的横、纵坐标值，m∈{0,...,M-1}，n∈{0,...,N-1}；
(4.2)用高斯函数分别标记各个捏合指尖候选样本pm,n；并采用公式(1)(2)、(3)和(4)计算各个捏合指尖候选样本pm,n的优化损失函数：
Figure PCTCN2016109614-appb-000008
w＝∑m,nα(m,n)φ(pm,n)            (2)
Figure PCTCN2016109614-appb-000009
Figure PCTCN2016109614-appb-000010
其中，φ是希尔伯特空间映射函数，λ是正则化参数，w是闭合形式的解，α是系数，F和F-1分别为离散傅里叶变换和离散傅里叶逆变换，y是回归函数标记图像块矩阵，*是共轭复数，p是用捏合指尖区域的CbCr-Gray-HOG特征表示的图像块矩阵，p'是特征图像块矩阵p的转置，pc是特征图像块矩阵p中第c维的特征图像块，σ2是高斯函数中的方差，⊙是傅里叶域的元素间相乘；
寻找公式(1)计算所得的最小值以获取损失最小值，得到损失最小值对应的系数α，将该系数α代入到公式(5)中：
Figure PCTCN2016109614-appb-000011
其中，z是捏合指尖搜索区域图像块，
Figure PCTCN2016109614-appb-000012
为核相关卷积后的响应值；
将各个捏合指尖候选样本pm,n分别代入到公式(5)中计算得到各个捏合指尖候选样本pm,n对应的
Figure PCTCN2016109614-appb-000013
将
Figure PCTCN2016109614-appb-000014
最小值对应的捏合指尖候选样本pm,n作为当前处理视频帧图的捏合指尖区域，捏合指尖候选样本pm,n对应的点(m,n)为当前处理视频帧图的捏合指尖区域中心点。
在第五步中，所述判断捏合指尖跟踪结束条件是否成立是指，设定阈值T3和T4，判断在若干帧连续的视频帧图中，相邻两帧视频帧图的捏合指尖区域中心点的横坐标变化值和纵坐标变化值是否分别小于T3和T4：若是，则捏合指尖跟踪结束条件成立，否则不成立。
本发明捏合指尖跟踪方法将捏合指尖在同一位置的小范围区域内持续停留一定时间作为判断跟踪结束的判断标准。
所述阈值T3设定为捏合指尖区域宽度的十分之一；所述阈值T4设定为捏合指尖区域高度的十分之一。
本发明捏合指尖跟踪方法应用于空中书写系统中，可准确识别捏合指尖运动轨迹，从而使捏合指尖空中书写系统得以实现；不同于单个指尖的空中书写方式，捏合指尖空中书写较为贴切地模拟了人手拿笔书写时的实际姿态，保留了手写轨迹的灵活性，使得基于视频的空中书写系统真实地反映实际的手写情况。
上述实施例为本发明较佳的实施方式，但本发明的实施方式并不受上述实施例的限制，其他的任何未背离本发明的精神实质与原理下所作的改变、修饰、替代、组合、简化，均应为等效的置换方式，都包含在本发明的保护范围之内。

Claims (9)

  1. 一种基于视频的捏合指尖跟踪方法，用于对空中书写中的捏合手指进行持续跟踪；其特征在于：包括以下五个步骤：
    第一步，捕获视频帧图，将视频帧图从RGB图像转化为YCbCr图像，通过YCbCr肤色检测器分割出肤色区域，采用拳头检测器对肤色区域进行分割得出拳头区域，在拳头区域中采用捏合指尖级联分类器定位捏合指尖区域，确定捏合指尖初始位置；
    第二步，捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样方法获取轮廓点；采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，确定肤色目标搜索区域的运动距离；
    第三步，构建捏合指尖肤色映射表；将上一帧视频帧图的捏合指尖区域划分为十六个区域单元，分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，结合各个区域单元的灰度值得到CbCr-Gray颜色特征；计算上一帧视频帧图捏合指尖区域的HOG特征，从而得到捏合指尖区域的CbCr-Gray-HOG特征；
    第四步，采用基于高斯核函数的滤波跟踪方法，结合所述捏合指尖区域的CbCr-Gray-HOG特征，确定当前处理视频帧图的捏合指尖区域中心点；
    第五步，判断捏合指尖跟踪结束条件是否成立：若成立，则跟踪结束；否则跳至第二步。
  2. 根据权利要求1所述的基于视频的捏合指尖跟踪方法，其特征在于：在第一步中，所述捕获视频帧图，将视频帧图从RGB图像转化为YCbCr图像，通过YCbCr肤色检测器分割出肤色区域，采用拳头检测器对肤色区域进行分割得出拳头区域，在拳头区域中采用捏合指尖级联分类器定位捏合指尖区域，确定捏合指尖初始位置包括以下步骤：
    (1.1)捕获若干帧连续的视频帧图；
    (1.2)将各帧视频帧图像分别从RGB图像转化为YCbCr图像；
    (1.3)根据人体肤色聚类特性，构造YCbCr肤色滤波器；采用YCbCr肤色滤波器确定各个YCbCr图像中的肤色区域，将非肤色区域剔除从而消除光照和非肤色区域的干扰；
    (1.4)离线训练基于Haar-like特征的拳头分类器，并用拳头分类器定位各个肤色区域中的拳头区域；
    (1.5)离线训练基于HOG特征的捏合指尖级联分类器，并用捏合指尖级联分类器定位各个拳头区域中的捏合指尖区域；
    (1.6)设定阈值T1和T2；判断各个捏合指尖区域中心点的横坐标变化最大值和纵坐标变化最大值：如果横坐标变化最大值≤T1，且纵坐标变化最大值≤T2，则设定所述若干帧连续的视频帧图中最后一帧视频帧图的捏合指尖区域中心点为捏合指尖初始位置，并跳至第二步；否则捕获后续若干帧连续的视频帧图，并跳至步骤(1.2)。
  3. 根据权利要求2所述的基于视频的捏合指尖跟踪方法，其特征在于：在步骤(1.3)中，所述根据人体肤色聚类特性，构造YCbCr肤色滤波器是指，将YCbCr图像进行Y通道、Cb通道和Cr通道分离，选定Cb通道数值在色带范围[77,127]内且Cr通道数值在色带范围[133,173]内的区域作为肤色区域。
  4. 根据权利要求2所述的基于视频的捏合指尖跟踪方法，其特征在于：在步骤(1.6)中，所述阈值T1设定为捏合指尖区域宽度的十分之一；所述阈值T2设定为捏合指尖区域高度的十分之一。
  5. 根据权利要求1所述的基于视频的捏合指尖跟踪方法，其特征在于：在第二步中，所述捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样方法获取轮廓点；采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，确定肤色目标搜索区域的运动距离是指：包括以下步骤：
    (2.1)捕获下一帧视频帧图作为当前处理视频帧图；以上一帧视频帧图获 得的捏合指尖区域中心点为中心在当前处理视频帧图上建立目标搜索区域；采用YCbCr肤色检测器提取目标搜索区域的肤色区域，从而获得当前处理视频帧图的肤色目标搜索区域；
    (2.2)采用Canny边缘检测器分别对上一帧视频帧图的肤色目标搜索区域和当前处理视频帧图的肤色目标搜索区域进行处理获取轮廓，并采用密集采样方法获取轮廓点；
    (2.3)采用LK光流法对上一帧视频帧图和当前处理视频帧图的轮廓点进行匹配，对上一帧视频帧图和当前处理视频帧图中各个对应轮廓点分别计算出运动距离，将运动距离的均值或中间值定义为肤色目标搜索区域的运动距离。
  6. 根据权利要求1所述的基于视频的捏合指尖跟踪方法，其特征在于：在第三步中，所述构建捏合指尖肤色映射表；将上一帧视频帧图的捏合指尖区域划分为十六个区域单元，分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，结合各个区域单元的灰度值得到CbCr-Gray颜色特征；计算上一帧视频帧图捏合指尖区域的HOG特征，从而得到捏合指尖区域的CbCr-Gray-HOG特征是指：包括以下步骤：
    (3.1)基于人体肤色的聚类特性，选择十六种不同色带的肤色构成捏合指尖肤色映射表；
    (3.2)将上一帧视频帧图的捏合指尖区域划分为十六个区域单元并映射到肤色映射表中，以分别寻找各个区域单元CbCr色带在捏合指尖肤色映射表中最接近的数值，分别计算各个区域单元的灰度值，得到十六维捏合指尖区域的CbCr-Gray颜色特征；
    (3.3)计算上一帧视频帧图捏合指尖区域在九个梯度方向上的HOG特征，每个方向分别包括三维，共得到二十七维HOG特征，结合五维基础HOG特征，并剔除梯度值为全零的一维冗余HOG特征，构成三十一维捏合指尖区域的HOG特征；
    (3.4)十六维捏合指尖区域的CbCr-Gray颜色特征与三十一维捏合指尖区域的HOG特征结合得到四十七维捏合指尖区域的CbCr-Gray-HOG特征。
  7. 根据权利要求6所述的基于视频的捏合指尖跟踪方法，其特征在于：在第四步中，所述采用基于高斯核函数的滤波跟踪方法，结合所述捏合指尖区域的CbCr-Gray-HOG特征，确定当前处理视频帧图的捏合指尖区域中心点是指： 包括以下步骤：
    (4.1)将上一帧视频帧图的捏合指尖区域中心点偏移所述肤色目标搜索区域的运动距离后得到的点作为中心在当前处理视频帧图上划定采样区域；在采样区域中，以上一帧视频帧图的捏合指尖区域中心点为中心建立大小为M×N的捏合指尖搜索区域；用循环矩阵对捏合指尖搜索区域进行循环采样得到若干捏合指尖候选样本pm,n，其中m、n分别是指捏合指尖候选样本pm,n的横、纵坐标值，m∈{0,...,M-1}，n∈{0,...,N-1}；
    (4.2)用高斯函数分别标记各个捏合指尖候选样本pm,n；并采用公式(1)(2)、(3)和(4)计算各个捏合指尖候选样本pm,n的优化损失函数：
    Figure PCTCN2016109614-appb-100001
    w＝∑m,nα(m,n)φ(pm,n)               (2)
    Figure PCTCN2016109614-appb-100002
    Figure PCTCN2016109614-appb-100003
    其中，φ是希尔伯特空间映射函数，λ是正则化参数，w是闭合形式的解，α是系数，F和F-1分别为离散傅里叶变换和离散傅里叶逆变换，y是回归函数标记图像块矩阵，*是共轭复数，p是用捏合指尖区域的CbCr-Gray-HOG特征表示的图像块矩阵，p'是特征图像块矩阵p的转置，pc是特征图像块矩阵p中第c维的特征图像块，σ2是高斯函数中的方差，⊙是傅里叶域的元素间相乘；
    寻找公式(1)计算所得的最小值以获取损失最小值，得到损失最小值对应的系数α，将该系数α代入到公式(5)中：
    Figure PCTCN2016109614-appb-100004
    其中，z是捏合指尖搜索区域图像块，
    Figure PCTCN2016109614-appb-100005
    为核相关卷积后的响应值；
    将各个捏合指尖候选样本pm,n分别代入到公式(5)中计算得到各个捏合指尖候选样本pm,n对应的
    Figure PCTCN2016109614-appb-100006
    将
    Figure PCTCN2016109614-appb-100007
    最小值对应的捏合指尖候选样本pm,n作为当前处理视频帧图的捏合指尖区域，捏合指尖候选样本pm,n对应的点(m,n)为当前处理视频帧图的捏合指尖区域中心点。
  8. 根据权利要求1所述的基于视频的捏合指尖跟踪方法，其特征在于：在 第五步中，所述判断捏合指尖跟踪结束条件是否成立是指，设定阈值T3和T4，判断在若干帧连续的视频帧图中，相邻两帧视频帧图的捏合指尖区域中心点的横坐标变化值和纵坐标变化值是否分别小于T3和T4：若是，则捏合指尖跟踪结束条件成立，否则不成立。
  9. 根据权利要求8所述的基于视频的捏合指尖跟踪方法，其特征在于：所述阈值T3设定为捏合指尖区域宽度的十分之一；所述阈值T4设定为捏合指尖区域高度的十分之一。
PCT/CN2016/109614 2016-10-25 2016-12-13 一种基于视频的捏合指尖跟踪方法 Ceased WO2018076484A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
SG11201900381PA SG11201900381PA (en) 2016-10-25 2016-12-13 Method for tracking pinched fingertips based on video

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610933351.7 2016-10-25
CN201610933351.7A CN106408579B (zh) 2016-10-25 2016-10-25 一种基于视频的捏合指尖跟踪方法

Publications (1)

Publication Number Publication Date
WO2018076484A1 true WO2018076484A1 (zh) 2018-05-03

Family

ID=58011921

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/109614 Ceased WO2018076484A1 (zh) 2016-10-25 2016-12-13 一种基于视频的捏合指尖跟踪方法

Country Status (3)

Country Link
CN (1) CN106408579B (zh)
SG (1) SG11201900381PA (zh)
WO (1) WO2018076484A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110765814A (zh) * 2018-07-26 2020-02-07 杭州海康威视数字技术股份有限公司 一种黑板书写行为识别方法、装置及摄像机
CN112861641A (zh) * 2021-01-15 2021-05-28 复旦大学 一种面向人机交互领域的动态手势识别硬件加速器
CN113610888A (zh) * 2021-06-29 2021-11-05 南京信息工程大学 一种基于高斯平滑的孪生网络目标跟踪方法
CN115171158A (zh) * 2022-07-05 2022-10-11 齐鲁工业大学 一种基于视觉特征的指尖检测方法

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107240122A (zh) * 2017-06-15 2017-10-10 国家新闻出版广电总局广播科学研究院 基于时空连续相关滤波的视频目标跟踪方法
CN109448189A (zh) * 2018-10-29 2019-03-08 极鱼（北京）科技有限公司 无人银行管理办法、无人银行和无人银行系统
CN111931722B (zh) * 2020-09-23 2021-02-12 杭州视语智能视觉系统技术有限公司 一种结合颜色比率特征的相关滤波跟踪方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102819733A (zh) * 2012-08-09 2012-12-12 中国科学院自动化研究所 一种快速的街景图像中人脸的检测模糊方法
EP2840527A1 (en) * 2013-08-22 2015-02-25 Fujitsu Limited Image processing device and image processing method
US20150110347A1 (en) * 2013-10-22 2015-04-23 Fujitsu Limited Image processing device and image processing method
CN105261038A (zh) * 2015-09-30 2016-01-20 华南理工大学 基于双向光流和感知哈希的指尖跟踪方法
CN105335711A (zh) * 2015-10-22 2016-02-17 华南理工大学 一种复杂环境下指尖检测方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101901350B (zh) * 2010-07-23 2012-05-16 北京航空航天大学 一种基于特征向量的静态手势识别方法
CN104821010A (zh) * 2015-05-04 2015-08-05 清华大学深圳研究生院 基于双目视觉的人手三维信息实时提取方法及系统
CN105739702B (zh) * 2016-01-29 2019-01-22 电子科技大学 用于自然人机交互的多姿态指尖跟踪方法
CN105975934B (zh) * 2016-05-05 2022-03-04 中国人民解放军63908部队 一种用于增强现实辅助维修的动态手势识别方法及系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102819733A (zh) * 2012-08-09 2012-12-12 中国科学院自动化研究所 一种快速的街景图像中人脸的检测模糊方法
EP2840527A1 (en) * 2013-08-22 2015-02-25 Fujitsu Limited Image processing device and image processing method
US20150110347A1 (en) * 2013-10-22 2015-04-23 Fujitsu Limited Image processing device and image processing method
CN105261038A (zh) * 2015-09-30 2016-01-20 华南理工大学 基于双向光流和感知哈希的指尖跟踪方法
CN105335711A (zh) * 2015-10-22 2016-02-17 华南理工大学 一种复杂环境下指尖检测方法

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110765814A (zh) * 2018-07-26 2020-02-07 杭州海康威视数字技术股份有限公司 一种黑板书写行为识别方法、装置及摄像机
CN110765814B (zh) * 2018-07-26 2023-07-14 杭州海康威视数字技术股份有限公司 一种黑板书写行为识别方法、装置及摄像机
CN112861641A (zh) * 2021-01-15 2021-05-28 复旦大学 一种面向人机交互领域的动态手势识别硬件加速器
CN112861641B (zh) * 2021-01-15 2022-05-20 复旦大学 一种面向人机交互领域的动态手势识别硬件加速器
CN113610888A (zh) * 2021-06-29 2021-11-05 南京信息工程大学 一种基于高斯平滑的孪生网络目标跟踪方法
CN113610888B (zh) * 2021-06-29 2023-11-24 南京信息工程大学 一种基于高斯平滑的孪生网络目标跟踪方法
CN115171158A (zh) * 2022-07-05 2022-10-11 齐鲁工业大学 一种基于视觉特征的指尖检测方法

Also Published As

Publication number Publication date
SG11201900381PA (en) 2019-02-27
CN106408579A (zh) 2017-02-15
CN106408579B (zh) 2019-01-29

Similar Documents

Publication Publication Date Title
Raghuveera et al. A depth-based Indian sign language recognition using microsoft kinect
Mukherjee et al. Fingertip detection and tracking for recognition of air-writing in videos
Zheng et al. Recent advances of deep learning for sign language recognition
Arooj et al. Enhancing sign language recognition using CNN and SIFT: A case study on Pakistan sign language
WO2018076484A1 (zh) 一种基于视频的捏合指尖跟踪方法
CN106682598B (zh) 一种基于级联回归的多姿态的人脸特征点检测方法
Agrawal et al. A survey on manual and non-manual sign language recognition for isolated and continuous sign
CN108509839A (zh) 一种基于区域卷积神经网络高效的手势检测识别方法
CN112101208A (zh) 高龄老人特征串联融合手势识别方法及装置
CN106407958B (zh) 基于双层级联的面部特征检测方法
CN106709936A (zh) 一种基于卷积神经网络的单目标跟踪方法
CN108052884A (zh) 一种基于改进残差神经网络的手势识别方法
Kumar et al. 3D sign language recognition using spatio temporal graph kernels
CN108171133A (zh) 一种基于特征协方差矩阵的动态手势识别方法
CN102930270A (zh) 基于肤色检测与背景消除的手部识别方法及系统
CN110096991A (zh) 一种基于卷积神经网络的手语识别方法
Hamed et al. Arabic sign language alphabet recognition based on HOG-PCA using microsoft kinect in complex backgrounds
CN107766864A (zh) 提取特征的方法和装置、物体识别的方法和装置
Nasri et al. A novel approach for dynamic hand gesture recognition using contour-based similarity images
Sruthi et al. Double-handed dynamic gesture recognition using contour-based hand tracking and maximum mean probability ensembling (MMPE) for Indian Sign Language
Kausar et al. LKECA-III: Novel localized keypoints extraction for sign language recognition using hidden Markov model
Liang et al. Egocentric hand pose estimation and distance recovery in a single RGB image
Fakhfakh et al. Gesture recognition system for isolated word sign language based on key-point trajectory matrix
Rady et al. Smart gesture-based control in human computer interaction applications for special-need people
CN110458064B (zh) 结合数据驱动型和知识驱动型的低空目标检测与识别方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16919918

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 19.08.2019)

122 Ep: pct application non-entry in european phase

Ref document number: 16919918

Country of ref document: EP

Kind code of ref document: A1