TW201508549A - Method for tracking and recording fingertip trajectory by image processing - Google Patents

Abstract

The present invention relates to a method for tracking and recording fingertip trajectory by image processing. Primarily, preprocessing image binarization for skin-tone colors can be carried out on a basis of image processing first, and then determine hand positions and find out fingertips by the characteristics that the positions of fingertips are usually on the external frame of a hand block. In such a case, the identified fingertip trajectory can be obtained for tracking and recording purposes.

Description

Image processing method for tracking and recording fingertip trajectories

The invention relates to an image processing method for tracking and recording fingertip trajectories, in particular to an innovative method for tracking and recording fingertips by means of image processing.

In recent years, in various consumer electronic products, touch devices with intuitive input and handwriting as input interfaces have gradually become popular. Since touch handwriting functions need to be achieved through stylus, handwriting or touch screen, The control device is usually limited to the size of the device itself, and the construction of the touch device requires a certain manufacturing cost. Therefore, in view of the convenience of handwriting input, the video lens is used to capture a continuous picture of the handwritten track, and then tracked by the image track line. The application of handwritten character recognition has been proposed. For example, Taiwan Patent Publication No. 466438 proposes a "method of constructing a gesture mouse", but it is mainly used in a simple environment, by means of a finger or other appropriate item in the photography area. Image, the method of operating the computer; or, as in Taiwan Patent Publication No. 393629, a "gesture recognition system and method and memory medium", but its main focus is on the gesture recognition system rather than the tracking system; A method of "calibrating gesture features in an image" is proposed in I224288, which first reveals Taking an input gesture image and performing image pre-processing on the gesture image to segment a closed curve formed by the binarized contour image of the gesture image, and then the image processing device can draw the gesture image according to the closed curve The curvature maps the spatial image, and then performs the convolution integral operation on the coordinate-peak group sequence formed by the coordinate group to specify the coordinate value having the largest integral value as the reference point, thereby obtaining the characteristic parameter of the gesture image. Finally, the feature parameters of the gesture image and the feature parameters of the reference gesture shape are compared to identify the gesture shape corresponding to the gesture image. However, the operation method is still too complicated, and how to capture and track the gesture is not discussed.

The reason is that the inventor has been rich in design and development and practical production experience of the relevant industry for many years, and has researched and improved the existing gesture recognition method to provide an image processing method for tracking and recording fingertip trajectory, in order to achieve better practical value. The purpose of the person.

The image processing method for tracking and recording fingertip trajectory of the present invention mainly comprises:

(A) Image skin color binarization processing: used to determine and binarize the color image block of the captured image data;

(B) Hand block determination: used to pick up the correct hand block, including:

(B1) skin color block calibration;

(B2) hand block screening;

(C) Fingertip coordinate tracking: By detecting the position of the fingertip, the fingertip coordinates are obtained; the steps include:

(C1) calculating the outer block of the hand block;

(C2) rim point coordinate search of the outer frame boundary;

(C3) Fingertip condition screening;

(D) Fingertip trajectory coordinate record: Each fingertip coordinate obtained is recorded to obtain a continuous fingertip trajectory.

The image processing method for tracking and recording the fingertip trajectory as described above, wherein in the step (B1), the connected pixel region calibration is used to find the connected pixels belonging to the same object in the image, thereby calibrating each of the images A skin-independent object that counts the number, area, and range of objects of all skin-tone objects in the image.

The image processing method for tracking and recording the fingertip trajectory as described above, wherein in the step (B1), after the entire image is calibrated by the connected element region, the first two skin color patches are further filtered by setting an area threshold value.

The image processing method for tracking and recording a fingertip trajectory as described above, wherein in the step (B1), the area threshold is 1200 pixels.

An image processing method for tracking and recording a fingertip trajectory as described above, wherein in step (B2), a skin color value between a minimum of 4 degrees and a maximum value of 25 degrees in a hue range is a red color. It is determined by the proportion of red skin in the image whether it is a hand.

The image processing method for tracking and recording the fingertip trajectory as described above, wherein, in the step (B2), when the ratio of the red skin color of the two major skin color blocks in the image is greater than a certain value, the larger ratio is the hand; If only one skin color block is larger than the red skin color ratio threshold, the block is a hand; if neither of them is inconsistent, the picture is determined to be no hand block.

The image processing method for tracking and recording the fingertip trajectory as described above, wherein in the step (C1), the hand block outer frame is calculated by moving the hand block pixels from left to right, from top to bottom. The coordinates of each point are counted to find the maximum and minimum values of the X and Y coordinates, and then the value of the rectangular frame is the outer frame of the hand block.

The image processing method for tracking and recording the fingertip trajectory as described above, wherein, in the step (C2), the outer frame of the hand block is sequentially searched, and the outer frame is a one-dimensional pixel, and the outer frame is four sides. Divided into four parts, starting from the four corners, searching sequentially in the clockwise direction. When there is a hand edge pixel on the outer frame, set this point as the midpoint of the fingertip Fm, and then use the fingertip midpoint Fm is the starting point X1, find the adjacent hand edge coordinates, after searching for the adjacent coordinates, set the origin to 0, and set the coordinate to the center X2 of the next search. After completing the first search in this way, Perform 30 times forward and backward to search for the front point Ff and the back point Fb to find the position of the cusp on the four sides of the frame.

The image processing method for tracking and recording the fingertip trajectory as described above, wherein in the step (C3), the fingertip position is selected from the four cusps on the four sides of the outer frame, and the cephalist F is first _m and the front and back points F _f and F _b calculate two vectors versus Then, the angle between the cusp coordinates and the 30-pixel coordinates of the front and rear heads can be obtained by calculating the angle between the two vectors, wherein: Then, the cusp with the smallest angle on the four sides is compared, and the sharp point with the smallest angle is the fingertip.

The image processing method for tracking and recording the fingertip trajectory as described above, wherein in the step (C3), when the angles on the four sides exceed 45 degrees, the included angle is determined to be a non-fingertip position.

For a more complete and clear disclosure of the technical content, the purpose of the invention and the effects achieved by the present invention, it is explained in detail below:

Please refer to the first figure, which is a flow chart of the image processing method for tracking and recording the fingertip trajectory of the present invention. The method of the invention comprises the following steps:

(A) Image skin color binarization:

First, the captured image data is color-converted to determine the skin color block, and then the skin color block is binarized [because the image skin color binarization processing of this part is a general prior art, and the invention is not claimed Focus, so I will not repeat them];

(B) Hand block determination:

The main reason is to take out the correct hand block from the image after binarization of the image skin color. Because in the general operating environment, the skin image captured by the lens does not have to be only the hand, usually the face will enter the mirror together, or the hand does not exist in the picture, so it is necessary to correctly mark the hand block to be correct. Find out where the fingertips are located. This process is mainly achieved by two steps: skin color block calibration and hand block screening.

(B1) Skin color block calibration: Each skin color block in the image in which the image skin color binarization is completed is marked, and the range, area, height and width of each block are calculated. In this step, the invention mainly uses the connected component labeling to find out the connected pixels belonging to the same object in the image, thereby calibrating the individual objects in the image, and when the whole image passes through the connected meta-region After calibration, you can count all the objects in the image, the area, the scope of the object and other information. In addition, after the entire image is calibrated by the connected element area, there are still some large noise points that cannot be ruled out, or the target object is away. The lens cannot be tracked too far. Therefore, the present invention filters the first two skin color blocks by setting an area threshold value, wherein the area threshold is 1200 pixels; entering the picture in the binarized image of the skin color In addition to the hand, the face may also exist at the same time, and usually the hand area will be smaller than the face. Therefore, after selecting the first two blocks that meet the threshold value, the face must be removed. Please refer to the second. The rectangular box in the figure shows the result of calibrating the first two skin color blocks in the image;

(B2) Hand block screening: Since the hand and face of the skin color binarized image may enter the picture at the same time, the hand block must be screened out. Usually the hand area is smaller than the face, and the skin of the palm is different from the back and face of the hand, and usually it is more white and pink than the face, and the proportion of this skin color in the hand block is much larger than the face. The block, therefore, the present invention determines whether the hand is the hand based on the proportion of the skin color. To calculate the skin color ratio, a threshold value is needed, and then the white and pink pixels in the skin color candidate block are counted, and the pixel corresponding to the threshold value is calculated. The ratio of the total pixel to the skin color block is called the red skin color ratio; wherein, the present invention is a red skin color with a skin color value between a minimum of 4 degrees and a maximum of 25 degrees in the hue range. After calculating the red skin color ratio of each block, the two skin color blocks can be compared. If the red skin color ratio of both blocks is greater than a certain value, the larger ratio is the hand; if only one block is larger than the red color For the proportional threshold, the block is the hand; if neither of them is inconsistent, the picture is judged as no hand block, please refer to the third picture (a) for the original image binarized by skin color, the third picture (b) Show that only the lip part of the face conforms to the red color block, and the hand has a higher coincidence rate in the palm. The third figure (c) is the hand block screening result, the white block is the hand, the gray is the Excluded face block;

(C) Fingertip coordinate tracking:

After the hand block is removed, the fingertip position detection can be started. The main method is to first calculate the outer frame of the hand block, and then perform the boundary point coordinate search from the outer frame, and finally determine the fingertip position from the boundary point coordinates to achieve the three steps;

(C1) Calculate the outer frame of the hand block: It has been found by experience that the fingertip of the index finger is usually located on the outer frame of the hand block in the manner of writing with the fingertip, so the fingertip position can be quickly obtained according to this phenomenon. The four figures show the position of the fingertip and the outer frame of the hand at each angle. The calculation method of the outer frame of the hand block is to count the pixel of the hand block from left to right, and count the coordinate position of each point from top to bottom. Find the maximum and minimum values of the X and Y coordinates, and then mark the rectangular frame with the value of the hand block;

(C2) Snap point coordinate search of the outer frame boundary: After calculating the outer frame of the hand block, the possible position of the fingertip is calculated. Since the finger will be on the outer frame of the hand block, it is from the four sides of the outer frame and the edge of the hand position. The search may be the position of the fingertip. Please refer to the fifth figure to reveal the schematic diagram of the fingertip search in the present invention. The outer frame is a one-dimensional pixel, so the four sides of the outer frame are divided into four parts, and the four corners are used as the starting point. The clock direction is searched sequentially. When there is a hand edge pixel on the outer frame, set this point to the midpoint of the fingertip Fm, and then use the fingertip midpoint Fm as the starting point X1, according to the sixth figure (a), (b) Find the adjacent hand edge coordinates in the order of A, B, C, D, E, F, G, and H. After searching for the adjacent coordinates, set the origin to 0 and set the coordinate to the next search. Center X2. After completing the first search in this way, perform 30 times of forward and backward respectively to search for the F _f and the back point F _{b of the} fingertip; when the four sides of the frame complete the search of the above steps, You can get the cusps of all the edges of the hand on the outer frame of the image and the 30-pixel coordinates of the front and the back of the cusp. The result is shown in the seventh figure, where the dotted triangle inside the hand edge is a sharp point and 30 pixels apart. Point drawn;

(C3) Fingertip Condition Screening: After finding the cusp position on the four sides of the image frame, then the fingertip position screening is performed from among the four sharp points to select the true fingertip position. As shown in the eighth figure, this step first calculates two vectors with the cusp F _m and the front and rear points F _f and F _b . versus Then, the angle between the cusp coordinates and the 30-pixel coordinates of the front and rear heads can be obtained by calculating the angle between the two vectors, wherein: ;

In the hand image, the angle between the fingertips in the four corners of the frame is the smallest, so the only point that matches the smallest angle on the four sides is the fingertip; but there is one exception, when the hand does not reach out. , the representative is not in the writing state, the fingertip does not exist, so the maximum value of the fingertip angle should be standardized. When the angle is exceeded, the angle is not the fingertip position, and the maximum angle of the fingertip is 45 degrees;

(D) Fingertip track coordinates record:

After the fingertip coordinates are determined, the fingertip trajectory can be recorded. The recording method of the invention is to identify the fingertip of each picture, and record when there is a fingertip in the picture, and not record if the fingertip condition is not met, as shown by the black coordinate point of the figure (a) of the ninth figure. After the recording is completed, the track status will be a standard sequence, as follows:

The ninth figure (a) coordinates are linked as shown in the ninth figure (b), wherein the ninth figure (a) is shown as the number 3 handwritten fingertip track coordinates, and the ninth figure (b) is shown Figure 9 (a) Figure 3 is a schematic diagram of the handwritten fingertip symmetry coordinates of the number 3; please also refer to the tenth figure, which is a schematic diagram of the handwritten fingertip symmetry of the English word s, by which the coordinate sequence can be used as a fingertip Track recognition.

In the present invention, after the image is captured by the video lens, after the skin color binarization process, the skin color block is first selected and the larger block is reserved, and then the hand block is filtered by the red skin color distribution ratio; Use the outer block of the hand block as the fingertip search range, search for all the cusp points on the frame that may be the hand, and then judge the fingertip position from the sharp point to continue the fingertip tracking and recording. The coordinates of the movement condition are recorded to make it a trajectory.

In addition, the present invention can further implement the recorded fingertip movement trajectory on the handwritten text input to achieve the operation mode of the image tracking fingertip as the input interface, and only one video lens is required for the hardware, and the cost is It will be much less than a touch device, so it has a strong market potential.

However, the above-described embodiments or drawings are not intended to limit the structure or the use of the present invention, and any suitable variations or modifications of the invention will be apparent to those skilled in the art.

In summary, the embodiments of the present invention can achieve the expected use efficiency, and the specific structure disclosed therein has not been seen in similar products, nor has it been disclosed before the application, and has completely complied with the provisions of the Patent Law. And the request, the application for the invention of a patent in accordance with the law, please forgive the review, and grant the patent, it is really sensible.

no

First: Flow chart of the present invention. Second figure: Schematic diagram of calibration result of skin color block of the present invention. Third figure: (a) original image binarized by skin color of the present invention; (b) red color pixel image of the present invention Schematic diagram; (c) hand block screening discrimination result of the present invention. FIG. 4 is a schematic view showing the position of the fingertip and the outer frame of the hand of the present invention at various angles. FIG. 5 is a schematic diagram showing the fingertip search situation of the present invention. 6: (a) revealing a schematic diagram of the anterior point search of the fingertip of the present invention; (b) revealing a schematic diagram of the search point of the fingertip of the present invention. FIG. 7 is a view showing the search result of the cusp position on the four sides of the image frame of the present invention. Figure 8: Schematic diagram of the calculation of the cusp angle of the hand of the present invention: (a) Schematic diagram of the handwritten fingertip trajectory coordinate point of the present invention; (b) The handwritten fingertip trajectory coordinate point of the present invention is connected to the state diagram of the tenth figure: The English word s handwriting fingertip trajectory coordinate point diagram of the present invention

Claims (10)

An image processing method for tracking and recording a fingertip trajectory, comprising: (A) image skin color binarization processing: for determining and binarizing the captured skin image data; (B) hand region Block decision: used to extract the correct hand block, including: (B1) skin color block calibration; (B2) hand block screening; (C) fingertip coordinate tracking: by detecting the fingertip position, Fingertip coordinates; the steps include: (C1) calculation of the hand block outer frame; (C2) outer frame boundary cusp coordinate search; (C3) fingertip condition screening; (D) fingertip track coordinate record: will be obtained Each fingertip coordinate is recorded to obtain a continuous fingertip trajectory. The image processing method for tracking and recording the fingertip trajectory described in claim 1 is characterized in that, in the step (B1), the connected pixel region calibration is used to find out the connected pixels belonging to the same object in the image, and borrow This calibrates the individual skin-individual objects in the image, and counts the number, area, and object range of all skin-color independent objects in the image. An image processing method for tracking and recording a fingertip trajectory as described in claim 2, wherein in step (B1), after the entire image is calibrated by the connected element region, the image is further filtered by setting an area threshold value. Two large skin blocks. An image processing method for tracking and recording a fingertip trajectory as described in claim 3, wherein in the step (B1), the area threshold is 1200 pixels. An image processing method for tracking and recording a fingertip trajectory as described in claim 4, wherein in step (B2), the color tone is between a minimum of 4 degrees and a maximum of 25 degrees. The value is red, and it is determined by the proportion of red skin in the image whether it is a hand. An image processing method for tracking and recording a fingertip trajectory as described in claim 5, wherein, in step (B2), when the proportion of the two major skin color patches in the image is greater than a certain value, the ratio is large. The hand is the hand; if only one skin color block is larger than the red skin color threshold, the block is the hand; if neither of them is inconsistent, the picture is determined to be no hand block. An image processing method for tracking and recording a fingertip trajectory according to claim 1, wherein in step (C1), the hand block outer frame is calculated by moving the hand block pixel from left to right. From top to bottom, the coordinates of each point are counted, and the maximum and minimum values of the X and Y coordinates are found. Then, the rectangular frame is the outer frame of the hand block. An image processing method for tracking and recording a fingertip trajectory according to claim 7, wherein in step (C2), the outer frame of the hand block is sequentially searched, and the outer frame is one-dimensional. Pixel, divide the four sides of the frame into four parts, starting from the four corners, and searching sequentially in the clockwise direction. When there is a hand edge pixel on the outer frame, set this point as the midpoint of the fingertip Fm, and then Using the fingertip midpoint Fm as the starting point X1, looking for the adjacent hand edge coordinates, after searching for the adjacent coordinates, the origin is set to 0, and the coordinate is set to the center X2 of the next search, and the method is completed in this way. After a search, 30 times of forward and backward are performed separately to search for the front point Ff and the back point Fb, so as to obtain the cusp position on the four sides of the outer frame. An image processing method for tracking and recording a fingertip trajectory according to claim 7, wherein in step (C3), fingertip position screening is performed from four cusps on four sides of the outer frame. First calculate the two vectors with the cusp F _m and the front and rear points F _f and F _b versus Then, the angle between the cusp coordinates and the 30-pixel coordinates of the front and rear heads can be obtained by calculating the angle between the two vectors, wherein: Then, the cusp with the smallest angle on the four sides is compared, and the sharp point with the smallest angle is the fingertip. An image processing method for tracking and recording a fingertip trajectory according to claim 9, wherein in the step (C3), when the angles on the four sides exceed 45 degrees, the angle is determined to be a non-finger position. .