TWI426420B - Hand track identification system and method thereof - Google Patents

Description

Hand trajectory identification system and method thereof

The invention relates to a hand trajectory identification system and a method thereof, in particular to having a hand video interface and using a chain code as a trajectory feature to re-use a confusion set for correction.

Currently used in handwriting technology, its conventional technology requires additional input devices, such as a tablet, a touch panel, such devices are relatively expensive, so that the cost of related products is increased; in addition, most of these technologies in the past A specific pen is needed as an input tool, and most of these pens are light and short, easy to lose, and cause a lot of inconvenience in operation.

Many touch technologies are now available for handwriting. However, these touch panels still have problems such as thick thickness, easy scratching of the contact surface, and poor light transmission. Although touch panels have gradually spread to many devices, such as smart phones, ATMs, and some touch-type personal computers, in the goal of computer home appliances, the touch panel is not practical; The cost of the control panel is relatively high and cannot be directly popularized in the market of large panels such as TV. Secondly, the touch interface of large panel applications is awkward because the user needs to operate between the seat and the panel.

Therefore, there is a need for a hand trajectory recognition system and method thereof that utilizes a remote operation similar to a touch interface to address the shortcomings of the prior art described above.

The main purpose of the present invention is to provide a hand track recognition system and method thereof, including a video camera module, a moving object detection module, a finger detection module, a feature capture module and a track recognition module, wherein the video camera The module captures the hand motion image, the moving object detection module integrates the background reconstruction and the foreground object color analysis technology to find the hand in the action, and the finger detection module locates the finger tip according to the separated hand image, and according to the finger The tip moves the position to record the trajectory, and the feature capture module uses the chain code as the trajectory feature. The trajectory identification module uses the radial basis function-like neural network to classify the chain code features, and then corrects the confusion by the confusion set to identify the finger writing. The symbol or text represented by the track.

Therefore, the present invention utilizes video signal detection to implement handwriting input, and can be used as a basis for computer actions, and performs computer control or text input, thereby improving the convenience of human-machine communication, and can be applied to computer home appliances and instant computer teaching. The human-machine interface required for office meetings, data entry, article writing or drawing, especially for patients with language impairments, can be directly written by finger without any external input device. Talk and communicate.

The embodiments of the present invention will be described in more detail below with reference to the drawings and the <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt;

Referring to the first figure, a schematic diagram of the hand trajectory recognition system of the present invention. As shown in the first figure, the hand track recognition system of the present invention includes a video camera module 10, a moving object detecting module 20, a finger detecting module 30, a feature capturing module 40, and a track recognition module 50. Used to provide a hand video interface for track recognition.

The video camera module 10 captures a picture to generate a plurality of continuous motion picture data T1, and the video camera module 10 may include a charge coupled device (CCD) or a complementary metal oxide half (CMOS) camera and has a plurality of pixels. Each motion picture data T1 includes a plurality of pixel values, and each pixel has a corresponding pixel value.

The moving object detecting module 20 receives the moving image data T1 to establish a background image, and simultaneously detects the moving object T2 in the moving image data T1. As the image moves and the time changes, the pixel values will also change synchronously. Therefore, the pixel values of the moving image data T1 are first counted to establish a histogram, and then the kernel function density estimation method (Kernel Density Estimation, KDE) for each pixel to generate an approximation corresponding pixel value distribution probability, and finally, according to the next moving image data T1, the probability of distribution of each pixel is determined to be a non-moving background or moving moving object T2, and The judgment criterion is a moving object T2 belonging to a desired one having a large distribution probability and belonging to the background and having a small distribution probability.

The finger detecting module 30 receives the moving object T2, first finds the fingertip, that is, the tip of the finger, and uses the color analysis technology to filter the moving object of the non-hand in the moving object T2, thereby finding the hand 32 in the action, and simultaneously finding the hand 32 in the action. The image of the hand 32 is separated, as shown in the second figure, and the outline of the hand image is framed by a Bounding Box 34, and the intersection 31, 34, 35 of the hand 31 and the surrounding rectangle 32 are overlapped. Centering, the angles of the contour lines before and after the intersection 33, 34, 35 are calculated individually, and the intersection point 33 having the smallest angle is the fingertip position, and finally the continuous position of the fingertip is connected to form the fingertip trajectory T3.

The feature capture module 40 receives the fingertip trajectory T3 representing the stroke of the written stroke to perform stroke extraction and writing feature extraction, and generates a trajectory feature T4. Stroke extraction is based on the writing characteristics. For example, when the stroke has a turn, a pen, or a pen down movement, the movement speed of the fingertip will be slowed down, and the fingertip trajectory T3 is estimated by the fingertip movement speed to take out the pause point, and then use the pause point. In order to distinguish the entry strokes in the fingertip trajectory T3, leave the strokes and the strokes, and delete the entry strokes and leave the strokes, only the written strokes are retained for judgment. The judgment method firstly encodes the written stroke in the N direction of the chain code according to the writing direction, N is a positive integer, and uses the histogram to count the probability of the moving direction of the stroke of the written stroke, and then evaluates the density of the kernel function. The test measures the probability distribution. Due to the cyclic coding characteristics of the chain code, the range of functions estimated by the kernel function estimation method will fall in a region smaller than 0 or more than N, so the probability of exceeding the range must be merged to its opposite end point, for example, for the direction. For the region of N+1, the corresponding probability must be added to the region of direction 0. This is called the cyclic chain probability distribution estimation, and the estimated probability function is used as the trajectory feature T4 for identification.

The trajectory identification module 50 receives the trajectory feature T4 and performs trajectory identification to generate the written symbol T5. The trajectory feature T4 is first classified according to a Radial Basis Function Neural Networks (RBFNN) to generate a preliminary Classification results. Since different characters or symbols in the trajectory feature T4 have similar chain code distributions, such as 0 and 0, 2 and Z, the Confusing Set is reused to correct the preliminary classification result and generate the required writing symbol T5. It can include handwritten symbols or handwritten words, which can be used as the basis for computer actions, thereby enabling computer control or text input.

Referring to the third figure, a schematic diagram of the method for identifying the hand trajectory of the present invention. As shown in the third figure, the hand trajectory identification method of the present invention begins with step S10. In step S10, a video camera module is used to capture a picture to generate a continuous plurality of motion picture data, each of which is dynamic. The image data includes a plurality of pixel values, and each pixel value corresponds to one pixel.

Then, in step S20, the moving object detection module is used to receive the dynamic image data, and the probability of pixel value distribution of each pixel is approximated by the kernel function density estimation method to generate a similar distribution probability and distributed. A pixel with a high probability belongs to the background, and a pixel with a small probability of distribution belongs to the moving object, thereby detecting the required moving object, and proceeds to step S30 to receive the moving object by using a finger detection module, and filtering by using color analysis technology. Move the non-hand moving object in the object to find the hand in motion, and then frame the outline of the hand image with the enclosing rectangle, and calculate the intersection before and after the intersection with the intersection of the hand and the enclosing rectangle. The angle of the line segment, and the intersection with the smallest angle is the position of the fingertip, connecting the continuous position of the fingertip to form the fingertip trajectory.

Then, in step S40, the fingertip trajectory is received by a feature extraction module, and the pause point is extracted by the estimation of the fingertip movement speed, and the pause point is used to distinguish the entry stroke in the fingertip trajectory T3 and leave the stroke. And writing strokes, the writing strokes are coded according to the writing direction of the N direction chain code (Chain Code), N is a positive integer, using the histogram to calculate the probability of moving the stroke direction of the stroke, and then using the kernel function density estimation The method estimates the probability distribution, and the estimated probability function is the trajectory feature.

Finally, proceeding to step S50, the trajectory identification module is used to receive the trajectory feature, and the trajectory features are first classified according to the radial basis function neural network to generate preliminary classification results, and then the confusion set is used to correct the preliminary classification result, and generate the The required writing symbol.

The video camera module, the moving object detecting module, the finger detecting module, the feature capturing module and the trajectory identifying module of the method of the present invention are the video camera module 10 and the moving object as described in the first figure. The detection module 20, the finger detection module 30, the feature extraction module 40, and the trajectory identification module 50 are therefore not described again.

The invention is characterized in that it is not limited by the input device, such as a touch panel, and the finger can be directly used as an input interface anytime and anywhere, so the applicability is strong. Another feature of the present invention is that, for the hand recognition technology, the present invention utilizes the minimum angle between the outline of the surrounding rectangle and the hand to locate the fingertip, and can improve the method of positioning the fingertip by generally recognizing the highest point of the hand. Thereby improving the accuracy of positioning. In addition, the present invention estimates the probability distribution of the chain code by using the kernel function estimation method as a trajectory feature, and classifies the neural network by using a radial basis function, and performs the output result through the established confusion set. Corrected to achieve a better recognition rate.

Therefore, the hand trajectory recognition system of the present invention can be used to control computer operation or text input, thereby improving the convenience of human-machine communication, and can be applied to computer home appliances, instant computer teaching, office meetings, data input, article writing. Or the human-machine interface required for drawing, especially for patients with language impairments, can use the finger writing method to communicate and communicate directly without any external input device.

The above is only a preferred embodiment for explaining the present invention, and is not intended to limit the present invention in any way, and any modifications or alterations to the present invention made in the spirit of the same invention. All should still be included in the scope of the intention of the present invention.

10. . . Video camera module

20. . . Moving object detection module

30. . . Finger detection module

31. . . hand

32. . . Enclosing rectangle

33. . . Intersection

34. . . Intersection

35. . . Intersection

40. . . Feature capture module

50. . . Trajectory identification module

S10. . . Using the video camera module to generate a continuous plurality of motion image data by taking a picture

S20. . . Using the moving object detection module to receive the dynamic image data borrowing function density estimation method to detect moving objects

S30. . . Using the finger detection module to receive the moving object to find the minimum angle intersection to form the fingertip trajectory

S40. . . Using the feature capture module to receive the fingertip trajectory for stroke extraction and writing feature capture to generate trajectory features

S50. . . Using the trajectory identification module to receive trajectory features for trajectory recognition to generate written symbols

T1. . . Motion picture data

T2. . . Moving object

T3. . . Fingertip track

T4. . . Trajectory feature

T5. . . Writing symbol

The first figure is a schematic diagram of the hand trajectory identification system of the present invention.

The second figure is a schematic diagram of finger detection according to the present invention.

The third figure is a schematic diagram of the method for identifying the hand trajectory of the present invention.

10. . . Video camera module

20. . . Moving object detection module

30. . . Finger detection module

40. . . Feature capture module

50. . . Trajectory identification module

T1. . . Motion picture data

T2. . . Moving object

T3. . . Fingertip track

T4. . . Trajectory feature

T5. . . Writing symbol

Claims (8)

A hand trajectory recognition system includes: a video camera module for taking a picture to generate a continuous plurality of motion picture data, each of the motion picture data comprising a plurality of pixel values, and each pixel value corresponds to a pixel; a moving object detecting module receives the moving image data, and counts the pixel values by a histogram, and then generates a pixel for each pixel by Kernel Density Estimation (KDE) The approximate pixel value of the approximation is distributed, and a judgment criterion is used to determine whether the pixel belongs to a non-moving background or a moving object; a finger detection module receives the moving object and first finds the fingertip. That is, the tip of the finger is used to filter the non-hand moving object in the moving object by using color analysis technology, thereby finding a hand in the action, and simultaneously separating the image of the hand, and then enclosing the hand with the rectangle. The outline of the image is framed and centered on an intersection of the hand and the enclosing rectangle to calculate an angle of the contour line segment before the intersection point, The intersection of the smallest angle is the position of the fingertip, and finally the continuous position of the fingertip is connected to form a fingertip track, thereby representing a writing stroke track; a feature capturing module, receiving the fingertip track, and performing strokes Extraction and writing feature extraction to generate a trajectory feature, first based on a writing feature and an estimate of the fingertip movement speed to extract the pause point, and then use the pause point to distinguish the entry stroke in the fingertip trajectory, leaving the stroke and writing Strokes, and deletes the strokes and leaves the strokes, leaving only the strokes for judgment. The judgment is to first encode the strokes in the N direction according to the writing direction, N is a positive integer, and borrows a histogram. Calculate the probability of moving the direction of the stroke of the stroke, and then use the kernel function density estimation method to estimate the probability distribution to generate the corresponding probability function as a trajectory feature for subsequent identification; and a trajectory identification module Receiving the trajectory feature and performing a trajectory identification to generate a written symbol based on a radial basis function neural network (Radial Basis Functi) On Neural Networks (RBFNN) classifies the trajectory features to produce a preliminary classification result, and then uses a Confusing Set to correct the preliminary classification result and generate the required written symbols. The hand trajectory recognition system of claim 1, wherein the video camera module comprises a charge coupled device (CCD) or a complementary MOS camera. According to the hand trajectory identification system of claim 1, wherein the mobile object detection module determines that the pixel with the high probability of belonging belongs to the background according to the next dynamic image data, and A pixel with a small distribution probability belongs to the moving object. According to the hand trajectory recognition system of claim 1, wherein the writing feature includes that the moving speed of the fingertip is slowed when the stroke has a turn, a pen, or a pen down motion. A hand trajectory identification method includes: using a video camera module to capture a picture, generating a continuous plurality of motion picture data, each of the motion picture data comprising a plurality of pixel values, and each pixel value corresponds to one Pixel; using a moving object detection module to receive the dynamic image data, using the histogram to count the pixel values, and then using the kernel function density estimation method to generate an approximation corresponding pixel value distribution probability for each pixel And using a judgment criterion to determine whether the pixel belongs to a non-moving background or a moving object; using a finger detection module to receive the moving object, first finding the fingertip, that is, the tip of the finger, using color The analysis technique filters the non-hand moving object in the moving object, and then finds one hand in the action, simultaneously separates the image of the hand, and then frames the outline of the hand with an enclosing rectangle, and The intersection of the hand and the enclosing rectangle is centered to calculate an angle of the contour line before the intersection, and the intersection with the smallest angle is The position of the fingertip is finally connected to the continuous position of the fingertip to form a fingertip trajectory, thereby representing a writing stroke track; receiving the fingertip trajectory by using a feature capturing module, and performing stroke drawing and writing feature extraction To generate a trajectory feature, first based on a writing feature and an estimate of the movement speed of the fingertip to take out the pause point, and then use the pause point to distinguish the entry stroke in the fingertip trajectory, leave the stroke and the stroke, and delete the stroke. And leaving the strokes, only retaining the strokes for judgment, and the judgment is to first encode the strokes in the N direction according to the writing direction, N is a positive integer, and the probability of moving the direction of the stroke of the stroke by the histogram is performed. Statistics, and then use the kernel function density estimation method to estimate the probability distribution to generate the corresponding probability function as a trajectory feature for subsequent identification; and use a trajectory identification module to receive the trajectory feature for a trajectory identification To generate a written symbol, the trajectory features are first classified according to a radial basis function neural network to generate a preliminary The result of the classification, and then the confusion set is used to correct the preliminary classification result and generate the required written symbols. The hand track recognition method according to claim 5, wherein the video camera module comprises a charge coupled device (CCD) or a complementary metal oxide half (CMOS) camera. According to the method for identifying a hand trajectory according to claim 5, wherein the criterion for determining the moving object detecting module is that, according to the next moving image data, the pixel with a large probability of distribution belongs to the background, and A pixel with a small distribution probability belongs to the moving object. According to the hand trajectory identification method described in claim 5, wherein the writing feature includes that the moving speed of the fingertip is slowed when the stroke has a turn, a pen, or a pen down motion.