TWI499937B - Remote control method and remote control device using gestures and fingers - Google Patents

Description

Remote control method using gesture and finger and remote control device

The invention relates to a remote control method, in particular to a remote control method and a remote control device using gestures and fingers.

The mouse and keyboard are the main tools for traditionally operating computers. With the evolution of electronic technology, the tools for operating computers have gradually evolved. For example, Wii uses gyroscope sensors to operate games, and Kinect uses structured light depth camera to detect The user's overall movement changes to control the game, and use the touch panel to control the phone or tablet.

Since the advancement of electronic technology, the traditional control methods have not been able to satisfy the public. The modern Human Computer Interface (HCI) is becoming more and more simple, intuitive and natural. There is a natural user interface (Natural). User Interface (NUI) is the interface that can interact with electronic products through the natural behavior (gesture or language) of the person. The biggest difference between NUI and traditional methods is that they are quite intuitive and do not need too much. The learning is easy to use and is suitable for the public of any age.

Taking gestures as an example, it can be divided into whether it needs to be in the user's hand. Two types of gloves with light source are worn on the camera, and then the image is processed by the camera and processed by the image to analyze the user's gesture and manipulate the computer. However, the glove is not required to analyze the gesture. Not precise enough, and limited to one-handed manipulation, so how to perform more detailed analysis and apply to the manipulation of both hands becomes a subject worthy of study.

Accordingly, it is an object of the present invention to provide a remote control method and a remote control device that utilize gestures and fingers that are applicable to both hands.

Therefore, the present invention utilizes a remote control method of a gesture and a finger, and is executed by a remote control device in conjunction with a connected computing device. The remote control device includes a memory and a processor connected to the memory, and the memory stores a code. The processor reads the code and executes the method to output a zoom control signal to control the size of an object executed by the computing device. The method includes the following steps:

(A) The processor sequentially receives a plurality of images from the computing device and performs subsequent steps for each image.

(B) The processor performs skin color detection on the two non-overlapping detection ranges in the image. If the processor detects a skin color region in the two detection ranges, the subsequent steps are performed.

(C) the processor separately performs morphological analysis on the two skin color regions, defining a palm block representing a palm of each skin color region and a number of finger blocks representing a single finger, and determining whether the number is equal to 2, If the number of finger blocks in each skin color area is equal to 2, follow-up step.

(D) the processor calculates the length of each finger block, and determines whether the ratio of the length of the finger block and the shorter length of the finger block in each skin color region is greater than a proportional threshold. If yes, the zoom control signal is calculated according to the position of the second-hand palm block and the zoom control signal is output to the computing device.

Preferably, in step (D), after determining whether the length ratio is greater than the proportional threshold, if the determination is yes, the value of a counter is incremented by one, otherwise the value of the counter is decremented by one, and the counter is greater than one. When the threshold is counted, the zoom control signal is calculated and output, otherwise, the process returns to step (A), wherein the counter is further provided with a count maximum value, and if the counter value is equal to the count maximum value, the value is no longer increased. .

Preferably, wherein the maximum value of the count is twice the threshold of the count.

Preferably, in the step (C), the morphological analysis is performed by performing erosion filtering on the two skin color regions, filtering out the area having too small area, obtaining the second hand palm block, and comparing each skin color block and corresponding The difference between the palm blocks is obtained by at least one finger possible area in each skin color block, and finally, the continuous skin color detection is performed in the possible area of the finger to obtain each finger block.

Preferably, in step (D), the palm center of each palm block is calculated first, and then the change of the distance of the second-hand palm center is calculated, and converted into the zoom control signal, and the calculation of each palm center is performed. The method is: when the palm of the hand is calculated for the first time or the previous time, the palm area is calculated. The geometric center of the block is used as the center of each of the palms; when the center of the palm has been calculated in the previous time, the center of the palm is used for the previous time, and a first direction scan is performed to find out the color of the skin region. a second boundary point in the first direction, and calculating a center point of the two boundary points, thereby finding a coordinate in the first direction of the center of each of the palms, and then performing a second direction sweep according to the result of the scanning in the first direction Sighting, finding the two boundary points in the second direction of the skin color region, and calculating the center point thereof, thereby finding the coordinates in the second direction of the center of the palms, the coordinates in the first direction and the second The coordinates in the direction are defined as the center coordinates of the palms, wherein the first direction is one of a horizontal direction and a vertical direction, and the second direction is the other of the horizontal direction and the vertical direction.

Therefore, the present invention utilizes a gesture and finger remote control device, including a memory and a processor connected to the memory, the memory stores a code, the processor reads and executes the code, and cooperates with a phase connection. The computing device performs the remote control method using gestures and fingers.

The effect of the invention is that: the invention can detect the shape of the finger by detecting the finger block and the palm block, and then analyze the length of the finger block to determine whether to output the zoom control signal, so that the user can conveniently Use the two-handed zoom control.

1‧‧‧Remote control

11‧‧‧ memory

12‧‧‧ Processor

2‧‧‧ camera

3‧‧‧ screen

31‧‧‧Detection range

4‧‧‧ Computing device

S1-S8‧‧‧ steps

S41-S47‧‧‧Steps

S71-S78‧‧‧Steps

P1‧‧‧ Previous Palm Center

P2‧‧‧ horizontal center point

P3‧‧‧ vertical center point

Other features and advantages of the present invention will be apparent from the following detailed description of the preferred embodiments of the accompanying drawings. FIG. 1 is a block diagram illustrating a preferred embodiment of the present invention utilizing a gesture and finger remote control method. example; 2 is a flow chart illustrating the preferred embodiment; FIG. 3 is a schematic diagram showing a screen display two detection ranges; and FIG. 4 is a schematic diagram illustrating a skin color detected in one of the detection ranges Figure 5 is a flow chart illustrating the sub-steps of step S4; Figure 6 is a schematic diagram illustrating the detected palm block; Figure 7 is a schematic diagram showing the nine squares painted around the skin color region, FIG. 8 is a schematic diagram illustrating a nine-square grid drawn around the palm block for finger scanning; FIG. 9 is a schematic diagram illustrating a vertical finger scan; FIG. 10 is a schematic diagram illustrating a horizontal direction finger scan; FIG. 11 is a schematic diagram illustrating the determined start and end points of each finger block; FIG. 12 is a schematic diagram illustrating the update manner of the center of the palm block; FIG. 13 is a schematic diagram illustrating the The state of use of the preferred embodiment; FIG. 14 is a flow chart illustrating the sub-steps of step S7; and FIG. 15 is a diagram illustrating the operation of a counter.

Referring to FIG. 1 to FIG. 3, the preferred embodiment of the present invention utilizes a remote control device for gestures and fingers. The remote control device 1 is implemented by a camera 2, a screen 3, and a computing device 4. The remote control device 1 includes a storage device. The memory 11 of the code and a processor 12 connected to the memory 11. In this implementation For example, the remote control device 1 is a digital chip, and is connected to the computing device 4 and connected to the computing device 4. The computing device 4 is connected to the camera 2 and the screen 3 respectively, but not limited thereto. It is also possible to compile the method into a program software, which is executed by the computing device 4, but may have a better computing speed in the form of a digital chip. The processor 12 of the remote control device 1 reads and executes the code to output a zoom control signal to the computing device 4 for controlling the size of an object (not shown) executed by the computing device 4, the object For example, a picture, a movie, a text or a software use interface, etc., the object itself is not the focus of the present invention, but only a controlled subject. The method includes the following steps:

Step S1: The processor 12 sequentially receives the plurality of images captured by the camera 2, and performs subsequent steps for each image.

Step S2: The processor 12 performs skin color detection on the two non-overlapping detection ranges 31 in the image. In detail, the processor 12 pre-defines the initial range size and location of the two detection locations, and transmits the image to the computing device 4, and the computing device 4 outputs the image captured by the camera 2 on the screen 3, and The image overlays the area where the two detection ranges 31 (see FIG. 13 in actual operation), and prompts the user to extend the two hands into the two detection ranges 31, and the processor 12 continues to The second detection range 31 performs skin color detection.

It should be noted that the processor 12 performs foreground detection and then performs skin color detection on the pixels in the foreground portion. In this embodiment, progressive background image generation is used (see: Progressive Background Image Generation). Li Zongxi, moving object detection based on two-dimensional realization of 3D video conversion technology and chip design, master's thesis, National Taipei University of technology, Taipei, 2012) foreground detection, and use C _B and C YC _B C _R color space _{The R} component is used to determine whether it is skin color (see Wu Yueming, He Hanwu, Ru Tong, and Zheng Detao, "Hand Segmentation for Augmented Reality System," Second Workshop on Digital Media and its Application in Museum & Heritages , Chongqing, China, December, 2007, pp. 395-401.), foreground detection and skin color detection are not the focus of the present invention and will not be described herein.

Step S3: If the processor 12 detects a skin color region (such as the reversed hand region in FIG. 4) in the two detection ranges 31, the subsequent steps are performed; otherwise, the process returns to step S1. In detail, the step further includes an initial step of performing two-hand detection, and the initial step is for confirming whether to start the subsequent step, and therefore the skin color region must continue to exist for a period of time. In this embodiment, the computing device 4 In addition to prompting the user to move his or her hands to the detection range 31 on the screen 3, the user is also prompted to "push" his hands forward to make a "push" action, and the processor 12 utilizes a finite state. The algorithm of the Fine State Machine detects whether the area of the skin color area is gradually increased first, then gradually decreases, and if so, the subsequent steps are performed. However, it is not limited to the above manner, and the subsequent steps can be performed immediately after the skin color region is detected.

Wherein, the finite state machine is used to detect that the area of the skin color region is continuously increased by three times or more than three times, and then continuously decreased by three times. At this time, it is determined that the pushing action has been made. It should be noted that the position of the detection range 31 may change after the action of the push is determined. When it is returned to this step, the detection of the push action in the above initial step is not performed, unless the detection range 31 is reset, refer to step S47 described later. Next, the next steps are explained.

Step S4 - The processor 12 performs morphological analysis on the two skin color regions respectively, and defines a palm block representing a palm of each skin color region and a number of finger blocks representing a single finger.

Referring to FIG. 5 and FIG. 6 , in detail, in this step, the action of removing the finger (step S41 ) is first performed by using a disconnect filter of a Morphological Filter mainly used in digital image processing ( Opening Filter) to filter out the smaller area of the finger relative to the palm. At this point, the palm block is obtained (as shown in the reverse part of FIG. 6), wherein the palm center of the palm block is also calculated (as shown in FIG. 6), and the calculation method of the palm center is described later.

Referring to Figures 5, 7, and 8, then, the possible area detection of the finger is performed (step S42), and the area where the finger is substantially located is found, in other words, the area where the finger is not located is excluded. The purpose of this step is to avoid a situation in which a false positive is generated in the case of the continuity detection described later. First, the center of the palm is centered, and the vicinity of the skin color area is divided into 3×3 nine-squares, so that the central E-square can be excluded first. Then, the skin color area and the palm block are compared, and the same part is excluded. In the example of FIGS. 7 and 8, the arm part of the H cell is excluded; thus, the only remaining skin color area falls in the B and D cells. In addition to avoiding subsequent misjudgments, it can also save computing time and improve efficiency.

Referring to Figures 5, 9, and 10, after finding out the possible areas of the finger, Finger scanning is performed on these possible finger regions (step S43), and the scanning is divided into horizontal scanning and vertical scanning in order to accommodate the fingers of the user at different rotation angles. First, a vertical scan of the continuous skin color is performed on the possible areas of the finger, and each pixel is judged to be the skin color one by one in the vertical direction, and the number of consecutive occurrences of the skin color pixels is calculated, and whether the number of consecutive skin colors is greater than the threshold value is determined. In order to eliminate the noise that is too short. For the hand skin color images of various angles, the results of (a) to (f) in Fig. 9 can be obtained. After the vertical scanning is finished, the horizontal scanning is performed in a similar manner, and the results of (a) to (f) in Fig. 10 can be obtained. It can be seen from the figure that the vertical scanning and the horizontal scanning have complementary defects. . Combine the two scan results and exclude the blocks with the number of pixels, which is the resulting multiple finger blocks.

Referring to Figures 5 and 11, finally, the start and end points of each finger block are calculated, wherein the start point is the point in the finger block closest to the center of the palm, and the end point is the point farthest from the center of the palm. In Fig. 11, the line connecting the starting point and the ending point is indicated, and the length of the line represents the distance between each starting point and the corresponding end point, and also represents the length of each finger block. Then it proceeds to step S5.

Referring to Figures 5, 12, and 13, it should be noted that, after the first action of removing the finger (step S41), or when the center of the palm is not calculated, the step calculates the geometric center of the palm block. The center of the palm is obtained, and when the center of the palm has been calculated in the previous time, the amount of calculation is reduced by tracking, which is based on the previous palm center P1 (Fig. 12(a)), first level Sweep the direction, find the left and right borders of the skin color area, and calculate the horizontal center point P2 to find the center of the palm The horizontal coordinate (Fig. 12(b)), then scan in the vertical direction according to the result of the horizontal scan, find the upper and lower boundaries of the skin color region, and calculate the vertical center point P3, thereby finding the center of the palm The vertical coordinate (Fig. 12(c)), the horizontal center coordinate and the vertical center coordinate, which is defined as the coordinates of the center of the palm (that is, P2).

After the palm center is continuously updated, the two detection ranges 31 are continuously updated (step S47), and the center of the two detection ranges 31 is moved to the new palm center coordinates, so that the skin color region can be continuously maintained. Falling within the detection range 31. It should be noted that the update of this step affects the action of step S3. In this embodiment, if at least one detection range 31 can no longer detect the skin color region, the detection range 31 is reset. Go back to the original reservation and wait for the user to "push".

Referring to Figures 2 and 13, step S5 is explained below.

Step S5 - The processor 12 calculates the number of finger blocks and determines that if the number of finger blocks of each skin color region is equal to 2, step S6 is performed.

Step S6 - The processor 12 calculates the length of each finger block, and proceeds to step S7.

Step S7 - The processor 12 determines whether the length ratio of the finger block having a longer length and a shorter length in each skin color region is greater than a proportional threshold. If yes, the process proceeds to step S8 to start the zoom function, otherwise Step S1. In this embodiment, the ratio threshold is 2, in other words, this step is to determine whether the long finger is more than twice the length of the short finger among the two fingers that are extended. This method is used to judge whether the extended finger is Forefinger and thumb, because in general, the gesture of two fingers that are easier to reach out, the length ratio will be doubled when the index finger and thumb are extended. Other gestures are not difficult to extend, it is the length of the finger. Closer.

Referring to Figures 14 and 15, further explanation, since the user does not necessarily maintain the angle of the finger and the camera 2 at any time, the ratio of the length of the two fingers in the image changes, and a mechanism for fault tolerance is required. Therefore, after determining whether the length ratio is greater than the proportional threshold (step S71), if the determination is YES, a counter is incremented by one (step S73), otherwise the counter is decremented by one (step S75), and the counter is greater than a count threshold. At the time (step S76), the zoom function is started in step S8 (step S77), otherwise it returns to step S1 (step S78). In addition, the counter is further provided with a count maximum value (step S72). After the counter value is excessively increased, it takes a long time to reflect that the length ratio is no longer greater than the proportional threshold. In this embodiment, the maximum value is This ratio is twice the threshold. Similarly, the counter should not be less than 0 (step S74) to immediately reflect the case where the length ratio is greater than the proportional threshold.

In this way, the value of the counter will be as shown in FIG. 15. First, when the correct gesture corresponding to the foregoing condition is detected, the value of the counter rises, exceeds the count threshold, and the zoom function is started, and then the value does not exceed the maximum value of the count. After the gesture disappears, the value of the counter drops, falls below the count threshold, turns off the zoom function (no longer proceeds to step S8), and falls to zero.

Referring to Figures 2 and 13, step S8 is explained below.

Step S8 - calculating and outputting the scaling control signal to control the size of the object executed by the computing device 4. The processor 12 is first remembered Record the current two palm center positions, calculate the distance between the two, and if it continues to be in the zoom function, that is, if you continue to enter this step, then when the distance becomes larger, calculate the output of an amplification control signal, on the contrary, If the distance becomes small, the output is reduced by a control signal.

The zoom control signal includes the amplification control signal and the reduction control signal. The manner of calculating the scaling control signal based on the two-point distance is well known to those skilled in the art and will not be described again. In this way, the processor 12 can zoom only when the user extends a specific gesture. If the user thinks that the zoom is not needed, it is convenient to not present the specific gesture.

In summary, the present invention can detect the shape of the finger by detecting the finger block and the palm block, and then analyze the length of the finger block to determine whether to output the zoom control signal, and have a fault tolerance mechanism judgment. The user can conveniently use both hands to perform the manipulation of zooming, so that the object of the present invention can be achieved.

The above is only the preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, that is, the simple equivalent changes and modifications made by the patent application scope and patent specification content of the present invention, All remain within the scope of the invention patent.

S1-S8‧‧‧ steps

Claims (3)

A remote control method using gestures and fingers is performed by a remote control device in conjunction with a connected computing device, the remote control device comprising a memory and a processor connected to the memory, the memory storing a code, the processor Reading the code to execute the method to output a zoom control signal to control the size of an object executed by the computing device, the method comprising the steps of: (A) the processor sequentially receiving a plurality of sheets from the computing device Image, and performing subsequent steps for each image; (B) the processor performs skin color detection on the two non-overlapping detection ranges in the image, if the processor is in the two non-overlapping detections A range of skin regions is detected in the range, that is, a total of two skin regions are detected, and then a subsequent step is performed; (C) the processor separately performs morphological analysis on the two skin regions, that is, respectively Erosion filtering is performed to partially filter out, and a palm block representing each palm area is defined, and then each skin color block and the corresponding palm block are compared, and each is obtained. At least one finger possible area in the color block, and finally continuous skin color detection in the possible area of the finger, and defining a number of finger blocks representing a single finger of each skin color area, and determining whether the quantity is equal to 2, if the number of finger blocks in each skin color region is equal to 2, that is, the processor defines a total of two palm blocks and four finger blocks for the two skin color regions, then performing the following steps; (D) The processor calculates the length of each finger block and determines each In the skin color region, whether the length of the finger block and the short length of the finger block are greater than a proportional threshold, if the determination is yes, the value of a counter is incremented by one, otherwise the counter is The value is decremented by one, and when the counter is greater than a count threshold, the zoom control signal is calculated according to the position of the second-hand palm block and the zoom control signal is output to the computing device, and when the counter is not greater than the count threshold, Then, the process returns to step (A), wherein the counter is further provided with a count maximum value. If the counter value is equal to the count maximum value, the value is no longer increased, and the count maximum value is twice the count threshold value. The remote control method using a gesture and a finger according to claim 1, wherein the step (D) is to first calculate a palm center of each of the palm blocks, and then calculate a change in the distance of the second-hand palm center, and convert the same to The control signal is scaled, and the calculation of each palm center is: when the first calculation or the previous calculation of the palm center is not performed, the geometric center of each palm block is calculated as the center of each palm; At the center of each palm, based on the previous palm center, a first direction scan is performed to find the two boundary points of the skin color region in the first direction, and the center point of the two boundary points is calculated. Thereby, finding the coordinates in the first direction of the center of the palm of the hand, and then scanning the second direction according to the result of the scanning in the first direction, finding the two boundary points in the second direction of the skin color region, and calculating a center point thereof, thereby finding a coordinate in the second direction of the center of the palm of the hand, the coordinate in the first direction and the coordinate in the second direction are defined as coordinates of the center of the palm of the hand, wherein The first The direction is one of a horizontal direction and a vertical direction, and the second direction is water The other direction is the flat direction and the vertical direction. A remote control device using gestures and fingers includes a memory and a processor connected to the memory, the memory stores a code, the processor reads and executes the code, and cooperates with a connected computing device A remote control method using a gesture and a finger as described in any one of claims 1 to 2 is performed.