WO2023004553A1

WO2023004553A1 - Method and system for implementing fingertip mouse

Info

Publication number: WO2023004553A1
Application number: PCT/CN2021/108497
Authority: WO
Inventors: 李德健
Original assignee: 广州视源电子科技股份有限公司
Priority date: 2021-07-26
Filing date: 2021-07-26
Publication date: 2023-02-02
Also published as: CN115885238A

Abstract

Provided in the present invention are a method and system for implementing a fingertip mouse, which are applied to a tablet computer. The method comprises: receiving an angle adjustment for an image collection apparatus of a tablet computer, such that the image collection apparatus is aligned with a finger activity area; acquiring a picture of the finger activity area by means of the image collection apparatus; acquiring, from the picture, the position and action of a fingertip of a target finger; determining the position of a fingertip mouse in a display interface of the tablet computer according to the position of the fingertip of the target finger; and according to the action of the fingertip of the target finger, controlling the fingertip mouse to execute, at the determined position, an operation corresponding to the action. By means of the present method, when using a tablet computer which is set up on a tabletop or a plane, a user can control an interface without lifting his/her arm, and the user can also arbitrarily change the position of a finger activity area by means of performing an angle adjustment on an image collection apparatus. The method can thus be adapted to even more different situations.

Description

Method and system for realizing fingertip mouse

technical field

The present invention relates to the field of fingertip mouse, more specifically, to a method and system for realizing the fingertip mouse.

Background technique

In recent years, tablet computers have gradually entered people's lives, bringing convenience to people in various aspects such as study, work, and entertainment. The reason why tablet computers are so popular is that apart from their small size and fast running speed, a large reason is the introduction of a new interactive method, that is, screen touch. flexible.

Certainly, PC computers have also made certain improvements, such as utilizing the mouse touchpad to replace the physical mouse, for users to utilize the touch of fingers to replace the mouse pointer in the physical mouse control interface, but the area of the touchpad on the laptop computer The location and size of the computer are limited. If the user is not sitting directly in front of the computer, it will be inconvenient to operate.

On the other hand, the tablet computer realizes the interactive mode of direct touch. The realization method is mainly to add a capacitive screen on the display screen. Determine what the user wants to do. However, this method also has certain defects. As shown in Figure 1, when the user stands up the tablet stand on the desktop for use, for example, when watching a video or using the tablet to take online classes, the user needs to lift it up when touching the capacitive screen. Arms, over time, the user's arms will feel tired.

If the tablet computer is improved with reference to the concept of the mouse touchpad of a PC computer, there will also be a problem that the position and size of the area of the touchpad are limited.

Contents of the invention

The present invention aims at overcoming at least one defect of the above-mentioned prior art, and provides a fingertip mouse implementation method and system for solving the problem that users cannot conveniently and quickly manipulate the tablet interface when using a tablet standing on a desktop.

The technical scheme adopted in the present invention comprises:

A method for implementing a fingertip mouse, which is applied to a tablet computer, comprising: receiving an angle adjustment of an image acquisition device of the tablet computer, so that the tablet image acquisition device is aligned with the finger active area; The device acquires the picture of the finger activity area; acquires the position and action of the fingertip of the target finger in the picture; determines the position of the fingertip mouse in the display interface of the tablet computer according to the position of the fingertip of the target finger position: controlling the fingertip mouse to perform an operation corresponding to the action at the determined position according to the action of the fingertip of the target finger.

The invention provides a method for implementing a fingertip mouse, which is applied to a tablet computer. The user can adjust the angle of the image acquisition device of the tablet computer manually or through an electric signal, so that the tablet image acquisition device is aligned with the finger activity area, and the finger activity area Generally, it is a desktop on which the tablet computer is set up or a plane that can be used to support the tablet computer. This method obtains the picture of the finger activity area through the image acquisition device of the tablet computer, acquires the fingertip of the target finger in the picture, and converts it into a fingertip mouse that can move and perform operations on the display interface of the tablet computer, specifically This is achieved by determining the position and movement of the fingertip. With the assistance of this method, the user can control the interface without raising the arm when using a tablet standing on a desktop or a plane. At the same time, the user can arbitrarily change the position of the finger active area by adjusting the angle of the image acquisition device of the tablet computer. , the method makes the user's use process more personalized and more adaptable to different situations. When the tablet image acquisition device does not need to be used to acquire finger activity areas, the device can also adapt to the user's usual use requirements for the device.

Further, before acquiring the position and motion of the fingertip of the target finger, the target finger is determined in the picture of the finger activity area according to a preset finger type.

The method provided by the present invention does not limit the type of the target finger, so when acquiring the fingertip position and movement of the target finger in the picture of the finger activity area, specifically, first determine the target in the picture of the finger activity area according to the preset finger type Which hand is the finger of the user? The user can set the type of finger applicable to this method by himself, which is more in line with his own needs.

Further, acquiring the position of the fingertip of the target finger in the screen is specifically: determining the probability that the fingertip of the target finger appears at each position in the screen, and placing the fingertip of the target finger The position with the highest probability of occurrence is taken as the position of the target finger.

When acquiring the position of the fingertip of the target finger in the screen, the probability of the fingertip of the target finger appearing in each position is used to determine the position with the highest probability of occurrence of the fingertip, and this position is used as the position of the target finger, so that the position of the fingertip of the target finger Positioning is more accurate.

Further, taking the position with the highest probability of occurrence of the fingertip of the target finger as the position of the target finger, specifically: judging whether the highest probability of occurrence of the fingertip of the target finger is greater than a preset position probability threshold, if so , taking the position with the highest probability of occurrence of the fingertip of the target finger as the position of the target finger.

When obtaining the position of the fingertip of the target finger in the screen, some objects may be misrecognized as the fingertip of the target finger, and the position that is misrecognized as the fingertip will also have a certain probability of the fingertip appearing, such as directly Taking the position with the highest probability of occurrence of the fingertip as the position of the fingertip may lead to the following misjudgment: in the picture where the fingertip of the target finger does not appear, some objects that are more similar to the fingertip are determined as the fingertip There is a certain probability value at the position of the object, and this probability value happens to be the position with the highest probability, so it is misjudged as the position of the fingertip. In order to avoid sending in this situation, a position probability threshold is set. When the highest probability exceeds the position probability threshold, the position corresponding to the highest probability is considered to be the position of the fingertip of the target finger.

Further, when it is determined that the highest probability of occurrence of the fingertip of the target finger is less than or equal to the position probability threshold, the picture of the finger movement area is re-acquired by the image acquisition device.

When the highest probability is less than or equal to the position probability threshold, it means that the object at this position is most likely not a fingertip. Therefore, the image acquisition device reacquires the picture until the position of the fingertip is obtained, and the setting of the position probability threshold can be more accurate. Identify the position of the fingertip from the screen, and reduce the possibility of misjudgment.

Further, determining the position of the fingertip mouse in the display interface of the tablet computer according to the position of the fingertip of the target finger, specifically: mapping the position of the fingertip of the target finger to the position of the tablet computer The corresponding coordinates in the display interface are used as the coordinates of the fingertip mouse in the display interface.

The finger activity area in the screen and the interface have a mapping relationship between the position in the screen and the coordinates in the interface. Therefore, when the screen is acquired by the image acquisition device, the position of the fingertip in the screen can be obtained and the position can be mapped through the mapping relationship. To the corresponding coordinates in the interface, replace the coordinates of the fingertip mouse displayed in the interface with the corresponding coordinates, thereby realizing that the position of the fingertip mouse moves with the position of the fingertip.

Further, the action of acquiring the fingertip of the target finger in the picture is specifically: acquiring a number of frames of the picture before and after the corresponding moment of the picture through the image acquisition device; according to the picture and the According to the several frames of pictures acquired, the action of the fingertip of the target finger is classified into the click behavior category or the non-click behavior category; Executing the operation corresponding to the action at the position, specifically: when the action of the fingertip of the target finger is classified into the click behavior category, controlling the fingertip mouse to perform a click operation at the corresponding coordinate; When the action of the fingertip of the target finger is classified into the non-click behavior category, the fingertip mouse is controlled to perform a non-click operation at the corresponding coordinates.

Since the movement of a fingertip is dynamic and cannot be reflected in a single static picture, it is necessary to obtain several frames of pictures before and after the moment of the picture through the image acquisition device to form a group of dynamic pictures, and obtain the fingertips more accurately through the dynamic pictures. sharp action. Since the mouse behavior categories are generally divided into click and non-click, the fingertip actions will be classified into the click behavior category and the non-click behavior category through the dynamic screen in advance, and then the fingertip mouse will be controlled correspondingly through the determined behavior category. Click or non-click operations are performed at the corresponding coordinates of the interface.

Further, classifying the action of the fingertip of the target finger into the click behavior category or the non-click behavior category is specifically: determining the probability that the action of the fingertip of the target finger is classified into the click behavior category, and classifying it into the click behavior category. The probability of the non-click behavior category; the probability that the action of the fingertip of the target finger is classified into the click behavior category, and the probability of being classified into the non-click behavior category are respectively compared with a preset classification probability threshold, and the The action of the fingertip of the target finger is classified into a click behavior category or a non-click behavior category corresponding to a probability greater than the classification probability threshold.

The classification of the fingertip action into the click behavior category or the non-click behavior category is also determined by probability. First, the probability of being classified into the two categories is determined, and then the probability of being classified into the two categories is compared with the classification probability threshold. The category corresponding to the probability greater than the classification probability threshold is taken as the category classified by the fingertip action.

A system for implementing a tablet fingertip mouse, comprising: a camera processing module, configured to receive an angle adjustment to an image acquisition device of a tablet computer, so that the image acquisition device is aligned with the finger activity area; The image acquisition device acquires the picture of the finger activity area at the current moment; the finger acquisition module is used to acquire the position and action of the fingertip of the target finger in the picture acquired by the image acquisition module; The position of the fingertip of the target finger acquired by the finger acquisition module determines the position of the fingertip mouse in the display interface of the tablet computer; The action of the fingertip controls the fingertip mouse to perform an operation corresponding to the action at the determined position.

Further, the finger acquisition module is further configured to determine the target finger according to a preset finger type before acquiring the position and movement of the fingertip of the target finger.

Compared with prior art, the beneficial effect of the present invention is:

The method and the system provided by the invention can enable the user to control the interface without raising the arm when using the tablet computer erected on the desktop. Secondly, the user can also change the position of the finger active area by adjusting the angle of the image acquisition device of the tablet computer. This method makes the user's use process more personalized and more adaptable to different situations. When the image acquisition device does not need to use When acquiring the finger activity area, the device can also adapt to the user's usual requirements for using the device. Furthermore, the method and system provided by the present invention will not limit the type of the target finger recognized in the screen, but will determine which hand the target finger belongs to according to the user or any externally preset finger type, and the user can set it by himself or herself. Determine the type of finger suitable for this method, which is more in line with your own needs.

Description of drawings

FIG. 1 is a schematic diagram of actions of a user manipulating a tablet when the tablet is stood on a table in the background art.

FIG. 2 is a schematic flowchart of methods S1 to S6 in an embodiment of the present invention.

FIG. 3 is a schematic diagram of actions of a user manipulating the tablet when the tablet is stood on the table in an embodiment of the present invention.

FIG. 4 is a schematic flowchart of method steps S31 to S32 in an embodiment of the present invention.

FIG. 5 is a schematic flowchart of method steps S321 - S322 in an embodiment of the present invention.

FIG. 6 is a schematic flowchart of method steps S3221 to S3222 in an embodiment of the present invention.

FIG. 7 is a schematic flowchart of method steps S41 - S42 in an embodiment of the present invention.

FIG. 8 is a schematic flowchart of method steps S51 - S52 in an embodiment of the present invention.

FIG. 9 is a schematic flowchart of method steps S521 to S525 in an embodiment of the present invention.

Fig. 10 is a schematic flow chart of steps T1-T15 in one of the preferred implementations of the embodiments of the present invention.

Fig. 11 is a schematic diagram of the module composition of the system in the embodiment of the present invention.

Detailed ways

The accompanying drawings of the present invention are only for illustrative purposes, and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some components in the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, some known structures and their descriptions in the drawings may be omitted. understandable.

This embodiment provides a method for implementing a fingertip mouse, which is applied to a tablet computer. A tablet computer generally refers to a small, portable personal computer, and an electronic device with a touch screen as a basic input. The touch screen (also known as digital tablet technology) that tablet computers have allows users to work with fingertips, stylus or digital pen instead of traditional keyboard or mouse. The fingertip mouse refers to using the fingertip simulation to control the mouse. After the tablet computer obtains information such as the fingertip simulation on the mouse control and the position of the fingertip, it renders and generates a mouse cursor on the display interface of the tablet computer. The mouse cursor will move correspondingly on the display interface and perform corresponding operations according to the obtained information such as the manipulation and position of the fingertip. The method provided in this embodiment can realize the aforementioned fingertip mouse in the display interface of the tablet computer.

As shown in Figure 2, the method provided in this embodiment includes the following steps S1-S6:

S1: receiving the angle adjustment of the image acquisition device of the tablet computer, so that the image acquisition device is aligned with the finger active area;

Specifically, the image capture device of the tablet computer can be a device suitable for capturing images such as a camera. As shown in FIG. Yes, the angle adjustment can be a manual adjustment by the user, or an adjustment signal sent by the user. After receiving the angle adjustment of the image acquisition device by the user, the image acquisition device can be aligned with the finger activity area. The finger activity area is generally the desktop on which the tablet is erected or any plane used to support the tablet computer. This area is provided for the user to move. The area where the finger performs fingertip mouse control. The finger activity area is not a fixed place, and the area targeted by the image acquisition device is the finger activity area, then the user can select an applicable area as the finger activity area by arbitrarily adjusting the angle of the image acquisition device. It can be seen that the method provided by this embodiment makes the user's use process more personalized and can adapt to different situations. usage requirements.

S2: Obtain the picture of the finger activity area through the image acquisition device; the image acquisition device aligns and photographs the finger activity area, and what is acquired in step S2 is at least one frame of video image captured by the image acquisition device.

S3: Obtain the position of the fingertip of the target finger in the picture of the finger activity area;

Specifically, as shown in Figure 4, step S3 includes the following steps:

S31: Determine the target finger in the picture of the finger activity area according to the preset finger type;

There may be more than one finger in the screen. Therefore, when determining the target finger in the screen, it is necessary to determine which finger in the screen is the target finger according to a preset finger type. Then, in the method provided by this embodiment, the user can pre-set the type of finger suitable for this method, and use the finger that is more suitable for his own use requirements as the target finger to manipulate the pointer mouse.

Specifically, before determining the target finger in the screen according to the preset finger type, it is necessary to identify the finger type of each finger in the screen. The method of realization may be: first use a certain number of training images about ten fingers The sample trains existing target detection models such as YOLO model, Mask R-CNN model, FCN (full convolutional network) model, etc., and obtains the trained target detection model. The trained target detection model can classify the image samples containing fingers, recognize the finger types of the fingers in the image samples, there are 10 types of fingers, corresponding to ten fingers, then in step S31, input the picture after training The target detection model, the target detection model can detect the finger types of all fingers appearing in the screen, then in step S31, the target finger can be determined in the screen according to the preset finger type, and its position and action can be determined in subsequent steps.

S32: Obtain the position of the fingertip of the target finger in the screen;

Specifically, as shown in Figure 5, obtaining the fingertip position of the target finger specifically includes the following steps:

S321: Determine the probability that the fingertip of the target finger appears at each position in the screen;

Since the finger type of the target finger has been determined, other fingers in the screen other than the target finger can be excluded, and no recognition or calculation is performed on other types of fingers. The specific implementation method of determining the probability of the fingertip of the target finger at each position in the screen can be: use a certain number of training image samples about the target finger to train the existing target detection model in advance, and obtain the trained target detection model . The trained target detection module can calculate the probability of the position where the image sample containing the target finger appears, so as to output the probability of each position of the target finger in the image sample.

S322: Taking the position with the highest probability of occurrence of the fingertip of the target finger as the position of the target finger;

Specifically, as shown in FIG. 6, the position with the highest probability of occurrence of the fingertip of the target finger is used as the position of the target finger, which specifically includes the following steps:

S3221: Determine whether the highest probability of the fingertip of the target finger appearing is greater than the preset position probability threshold, and when it is determined that the highest probability of the fingertip of the target finger appearing is greater than the position probability threshold, perform step S3222; When the highest probability of point occurrence is less than or equal to the position probability threshold, perform step S2 to reacquire the picture of the finger movement area through the image acquisition device;

When obtaining the position of the fingertip of the target finger in the screen, some objects may be misrecognized as the fingertip of the target finger, and the position that is misrecognized as the fingertip will also have a certain probability of the fingertip appearing, such as directly Taking the position with the highest probability of occurrence of the fingertip as the position of the fingertip may lead to the following misjudgment: in the picture where the fingertip of the target finger does not appear, some objects that are more similar to the fingertip are determined as the fingertip Point, a certain probability value appears at the position of the object, and the probability value is exactly the position with the highest probability, so it is misjudged as the position of the fingertip of the target finger. In order to avoid the occurrence of this situation, the position probability threshold is set. The position probability threshold refers to the probability that the fingertip of the target finger can be accurately determined to appear in a certain position. It is a preset value. The setting of the position probability threshold can The position of the fingertip can be recognized more accurately from the screen, and the possibility of misjudgment is reduced.

S3222: Taking the position of the fingertip of the target finger with the highest probability of appearing as the position of the target finger;

When the highest probability exceeds the position probability threshold, the position corresponding to the highest probability is confirmed as the position of the fingertip of the target finger. When the highest probability is less than or equal to the position probability threshold, it means that the object at this position is most likely not the fingertip of the target finger, and the image should be acquired again through the image acquisition device until the position of the fingertip is acquired.

S4: Determine the position of the fingertip mouse in the display interface of the tablet computer according to the position of the fingertip of the target finger;

As shown in Figure 7, the position of the fingertip mouse in the display interface of the tablet computer is determined according to the position of the fingertip of the target finger, specifically including the following steps:

S41: Map the position of the fingertip of the target finger to the corresponding coordinates in the display interface of the tablet computer;

There is a mapping relationship between the finger activity area in the screen and the position in the screen of the interface and the x, y coordinates in the interface. Specifically, the process of establishing the mapping relationship may be: first find the location of the screen in the screen captured by the image acquisition device Boundary, the size of the captured picture is determined by the picture boundary, and secondly, the size of the display interface of the tablet computer is also determined by the boundary, and the picture and the display interface are established according to the proportional relationship between the size of the picture and the size of the display interface mapping relationship. Specifically, if an xy coordinate system is established on the display interface, each point in the image can be mapped to each coordinate in the display interface directly according to the proportional relationship between the image and the display interface when establishing the mapping relationship.

S42: Use the corresponding coordinates as the coordinates of the fingertip mouse on the display interface of the tablet computer.

After obtaining the position of the fingertip in the screen, map the position to the corresponding coordinates in the interface through the mapping relationship, then replace the coordinates of the fingertip mouse displayed in the interface with the corresponding coordinates, thereby realizing the position of the fingertip mouse The pointed position moves.

S5: Obtain an action of the fingertip of the target finger in the screen;

The motion of the fingertip of the target finger refers to the dynamic motion made by the fingertip. Specifically, since the movement of the fingertip of the target finger is a dynamic movement, which cannot be reflected in a single static picture, it should be obtained through the combination of pictures of more than one frame, then correspondingly, the movement of the fingertip of the target finger is obtained in the picture When, specifically, several frame pictures of the video image picture before and after its corresponding time are obtained by the image acquisition device, and the combination of several frame pictures and the pictures obtained in the above steps can form a group of dynamic pictures, and the dynamic pictures can be more accurately Get actions at your fingertips. Preferably, 3 frames of pictures before and after the corresponding moment of the picture are acquired by the image acquisition device, a total of 6 frames of pictures. If the image acquisition device is in a state of just starting up, only 3 frames of pictures after the time corresponding to the picture are acquired.

In a specific implementation manner, as shown in FIG. 8, step S5 specifically includes the following steps:

S51: Obtain several frames of the video image before and after the corresponding moment through the image acquisition device;

S52: Classify the action of the fingertip of the target finger into the click behavior category or the non-click behavior category according to the picture and the acquired frames;

The behavior categories of fingertip actions can be divided into click behaviors and non-click behaviors. Click behaviors include at least fingertip simulated single-click or double-click actions, and non-click behaviors include at least fingertip simulated movements or drag-and-drop movements. action. Classify fingertip actions into click behavior category and non-click behavior category through the acquired dynamic picture, then the subsequent step can control the fingertip mouse to perform click or non-click operation in the interface according to the determined behavior category. Corresponding to click behaviors, fingertip mouse click operations include at least left or right click, left or right double click, etc. Corresponding to non-click behaviors, fingertip mouse non-click operations include at least dragging to move files, cursor movement, etc. . Optionally, the behavior category of the fingertip action can also be divided into any applicable behavior category according to different actual application scenarios, and each behavior category should have a corresponding execution operation of the fingertip mouse.

In an optional implementation manner, as shown in FIG. 9, step S52 includes the following steps:

S521: Determine the probability that the fingertip of the target finger is classified into the click behavior category and the probability that it is classified into the non-click behavior category according to the picture and the acquired frames;

The motion of the fingertip of the target finger can be classified by using an existing classification model such as a convolutional neural network classification model or other classification algorithms. Similar to training the target detection model, a certain number of training image samples of target fingertip movements can be used to train the existing classification model in advance, and the trained classification model can be obtained. The trained classification model can perform image samples containing target finger movements. Classification, the categories of classification are the click behavior category and the non-click behavior category. Specifically, the classification model will output the probability of classifying the action of the fingertip into the click behavior category or the non-click behavior category when processing the input frames and the pictures.

S522: Determine whether the probability that the action of the fingertip of the target finger is classified into the click behavior category is greater than the preset classification probability threshold, if yes, perform step S524; if not, perform step S523;

The classification probability threshold refers to a probability value that can accurately determine the category into which the fingertip action of the target finger is classified, and is a preset value. In fact, there is no order in judging whether the probability of being classified into the click behavior category or the non-click behavior category is greater than the classification probability threshold. This is just an optional implementation method. It is judged whether the probability of the behavior category is greater than the classification probability threshold, and then in step S523, it is judged whether the probability of being classified into the click behavior category is greater than the classification probability threshold.

S523: Determine whether the probability that the action of the fingertip of the target finger is classified into the non-click behavior category is greater than the preset classification probability threshold, if yes, perform step S525; if not, continue to perform step S521;

Generally speaking, using the trained classification model to classify fingertip actions will have a clear classification result, that is, the probability of only one classified category in the result will be higher than the classification probability threshold, indicating that the fingertip should be However, if there is a determination result that the probabilities of both categories are less than the preset classification probability threshold, you can choose to reclassify the several frames of pictures and the pictures input into the model, or This can be seen as an abnormal situation, and the classification model can be retrained to improve its accuracy, or the classification model can be replaced with a higher accuracy and larger classification model.

S524: Classify the action of the fingertip of the target finger into the click behavior category;

S525: Classify the motion of the fingertip of the target finger into a non-click behavior category.

S6: According to the movement of the fingertip of the target finger, the operation of controlling the fingertip mouse to perform a corresponding movement at the determined position.

Since the fingertip of the target finger is simulating the manipulation of the mouse, the action of the fingertip of the target finger can be to simulate the action of clicking, double-clicking or dragging and moving, then the corresponding position determined in the display interface of the tablet computer On the top, the fingertip mouse performs operations of left or right click, left or right double click, cursor movement or drag and drop to move files according to the fingertip movement of the target finger.

In a more specific implementation, as shown in Figure 10, the overall execution flow of the method provided in this embodiment is:

T1: Receive the angle adjustment of the image acquisition device of the tablet computer, so that the image acquisition device is aligned with the finger active area;

T2: Obtain the picture of the finger activity area through the image acquisition device;

T3: Determine the target finger in the picture of the finger activity area according to the preset finger type;

T4: Determine the probability that the fingertip of the target finger appears at each position in the picture;

T5: Determine whether the highest probability of the fingertip of the target finger appearing is greater than the preset position probability threshold, and when it is determined that the highest probability of the fingertip of the target finger appearing is greater than the position probability threshold, perform step T6; When the highest probability of point occurrence is less than or equal to the position probability threshold, perform step T2 to re-acquire the picture of the finger movement area through the tablet image acquisition device;

T6: take the position with the highest probability of the fingertip of the target finger as the position of the target finger;

T7: Map the position of the fingertip of the target finger to the corresponding coordinates in the display interface of the tablet computer, and use the corresponding coordinates as the coordinates of the fingertip mouse in the display interface;

After step T7 is executed for the first time to determine the corresponding coordinates of the target finger's fingertip mapping in the interface, T7 also includes rendering the pointer of the fingertip mouse at the corresponding position of the interface.

T8: Obtain several frames of pictures before and after the corresponding moment of the picture through the image acquisition device;

T9: According to the picture and several acquired frames, determine the probability that the fingertip of the target finger is classified into the click behavior category, and the probability of being classified into the non-click behavior category;

T10: Determine whether the probability that the action of the fingertip of the target finger is classified into the click behavior category is greater than the preset classification probability threshold, if yes, perform step T12; if not, perform step ST11;

T11: Determine whether the probability that the action of the fingertip of the target finger is classified into the non-click behavior category is greater than the preset classification probability threshold, if yes, perform step T13; if not, continue to perform step T9;

T12: classify the action of the fingertip of the target finger into the click behavior category, and perform step T14;

T13: classify the action of the fingertip of the target finger into the category of non-click behavior, and perform step T15;

T14: Control the fingertip mouse to perform a click operation at the corresponding coordinates;

T15: Control the fingertip mouse to perform non-click operations at the corresponding coordinates.

After completing steps T1 to T15 once, this method should be re-executed so that the fingertip mouse continues to move with the fingertip of the target finger in the interface and perform corresponding operations. If it is necessary to adjust the angle of the flat panel image acquisition device, start from step T1 starts to re-execute the method, if the angle of the tablet image acquisition device is no longer adjusted, then execute step T2 to continue to acquire the picture of the finger activity area through the tablet image acquisition device. Since this method is not executed for the first time, there is no need to render a new fingertip mouse pointer in the interface in step T7.

The method and system provided by this embodiment can enable the user to control the interface without raising the arm when using the tablet erected on the desktop or the plane, as shown in FIG. 3 , so that the user's arm will not feel tired. Secondly, the user can also change the position of the finger active area arbitrarily by adjusting the angle of the flat panel image acquisition device. This method makes the user's use process more personalized and more adaptable to different situations. Furthermore, the method provided in this embodiment does not limit the type of the target finger, and will determine which hand the target finger belongs to in the screen according to the user or any externally preset finger type, and the user can set the target finger by himself or herself. Determine the type of finger suitable for this method, which is more in line with your own needs.

Based on the same idea as the implementation method of the tablet computer fingertip mouse, as shown in FIG. 11 , this embodiment also provides a tablet computer fingertip mouse implementation system, including:

The camera processing module 100 is used to receive the angle adjustment of the image acquisition device of the tablet computer, so that the image acquisition device is aligned with the finger active area;

Specifically, the camera processing module 100 may be a software module, or may be a hardware module configured in the image capture device itself, such as a device or component for rotating the image capture device.

The image acquisition module 200 is used to acquire the picture of the finger activity area at the current moment through the image acquisition device;

The finger acquisition module 300 is used to acquire the position and action of the fingertip of the target finger in the picture acquired by the image acquisition module;

The finger analysis module 400 is used to determine the position of the fingertip mouse in the display interface of the tablet computer according to the position of the fingertip of the target finger; Execute the operation of the corresponding action at the position.

Specifically, the finger acquisition module 300 includes:

The finger determination sub-module 310 determines the target finger in the screen according to the preset finger type;

The finger acquisition submodule 320 is used to acquire the position and action of the fingertip of the target finger in the picture;

Specifically, the finger acquisition submodule 320 includes:

The position acquiring submodule 321 is used to acquire the position of the fingertip of the target finger;

Action acquisition sub-module 322, configured to acquire the action of the fingertip of the target finger;

Specifically, the location acquisition submodule 321 includes:

The first position acquisition subunit 3211 determines the probability that the fingertip of the target finger appears at each position in the screen;

The second position acquiring subunit 3212 takes the position with the highest probability of the fingertip of the target finger as the position of the fingertip of the target finger.

Specifically, when the second acquisition subunit 322 takes the position with the highest probability of occurrence of the fingertip of the target finger as the position of the target finger, the specific execution process is:

Judging whether the highest probability of the fingertip of the target finger appearing is greater than the preset position probability threshold, when it is determined that the highest probability of the fingertip of the target finger appearing is greater than the position probability threshold, the position of the highest probability of the fingertip of the target finger appearing As the position of the target finger; when it is determined that the highest probability of the fingertip of the target finger appearing is less than or equal to the position probability threshold, the picture of the finger movement area is re-acquired by the image acquisition device.

Specifically, the image acquisition module 200 includes:

The first picture acquisition subunit 210 is used to acquire the picture of the finger active area at the current moment through the image acquisition device;

The second picture acquisition subunit 210 is used to acquire several frames of pictures before and after the corresponding time of the picture through the image acquisition device;

Correspondingly, the action acquisition sub-module 322 classifies the action of the fingertip of the target finger as click Behavior categories or non-click behavior categories.

Specifically, the action acquisition submodule 322 includes:

The first action acquisition subunit 3221 is used to determine the probability that the action of the fingertip of the target finger is classified into the click behavior category, and the probability that it is classified into the non-click behavior category;

The second action acquisition subunit 3222 is used to compare the probability of the fingertip of the target finger being classified into the click behavior category, and the probability of being classified into the non-click behavior category with the preset classification probability threshold, and compare the target The motion of the fingertip of the finger is classified into the click behavior category or the non-click behavior category corresponding to the probability greater than the classification probability threshold.

Correspondingly, the finger analysis module 400 includes:

The finger position analysis sub-module 410 is configured to map the position of the fingertip of the target finger to the corresponding coordinates in the display interface of the tablet computer, and use the corresponding coordinates as the coordinates of the fingertip mouse in the display interface of the tablet computer.

The finger motion analysis sub-module 420 is used to control the fingertip mouse to perform a click operation at the corresponding coordinates of the mouse when the motion of the fingertip of the target finger is classified into the click behavior category; In the non-click behavior category, control the fingertip mouse to perform non-click operations at the corresponding coordinates.

In the implementation of the above-mentioned tablet fingertip mouse implementation system, the logical division of each functional module is only used as an example. In practical applications, the above-mentioned functions can be allocated according to needs, such as hardware configuration requirements or software implementation considerations. Completed by different functional modules, that is, the internal structure of the realization system of the tablet fingertip mouse can be divided into functional modules different from the above content, but can complete all the functions described above. Secondly, the implementation process of the modules of the tablet fingertip mouse implementation system of the above example is based on the same concept as the implementation method of the tablet fingertip mouse mentioned above in this embodiment, and its principle and technical effects are the same as those mentioned above. The implementation method of the tablet fingertip mouse is the same, and for specific content, please refer to the description of the method implementation mode, which will not be repeated here.

Apparently, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the technical solution of the present invention, rather than limiting the specific implementation manner of the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

A method for realizing a fingertip mouse, applied to a tablet computer, is characterized in that, comprising:

receiving an angle adjustment to the image acquisition device of the tablet computer, so that the image acquisition device is aligned with the finger active area;

Obtain the picture of the finger activity area through the image acquisition device; acquire the position and action of the fingertip of the target finger in the picture;

Determine the position of the fingertip mouse in the display interface of the tablet computer according to the position of the fingertip of the target finger;

controlling the fingertip mouse to perform an operation corresponding to the action at the determined position according to the action of the fingertip of the target finger.
The realization method of flat fingertip mouse according to claim 1, characterized in that, before the acquisition of the position and action of the fingertip of the target finger, it is determined in the picture of the finger activity area according to the preset finger type the target finger.
The implementation method of the tablet fingertip mouse according to claim 1, wherein acquiring the position of the fingertip of the target finger in the screen is specifically: determining that the fingertip of the target finger appears on the For the probability of each position in the screen, the position with the highest probability of occurrence of the fingertip of the target finger is taken as the position of the target finger.
The realization method of tablet fingertip mouse according to claim 3, is characterized in that,

The position with the highest probability of occurrence of the fingertip of the target finger is taken as the position of the target finger, specifically:

It is judged whether the highest probability of occurrence of the fingertip of the target finger is greater than a preset position probability threshold, and if so, the position of the highest occurrence probability of the fingertip of the target finger is taken as the position of the target finger.
The realization method of flat fingertip mouse according to claim 4, is characterized in that, also comprises:

When it is determined that the highest probability of the fingertip of the target finger appearing is less than or equal to the position probability threshold, the picture of the finger movement area is re-acquired by the image acquisition device.
The realization method of tablet fingertip mouse according to claim 1, is characterized in that,

Determine the position of the fingertip mouse in the display interface of the tablet computer according to the position of the fingertip of the target finger, specifically:

The position of the fingertip of the target finger is mapped to the corresponding coordinates in the display interface of the tablet computer, and the corresponding coordinates are used as the coordinates of the fingertip mouse in the display interface.
The realization method of tablet fingertip mouse according to claim 6, is characterized in that,

Acquiring the action of the fingertip of the target finger in the picture, specifically:

Acquiring several frame pictures of the picture before and after the corresponding moment by the image acquisition device; according to the picture and the acquired several frame pictures, the action of the fingertip of the target finger is classified into click behavior categories or non-click behavior categories;

According to the action of the fingertip of the target finger, the fingertip mouse is controlled to perform an operation corresponding to the action at the determined position, specifically:

When the action of the fingertip of the target finger is classified into the click behavior category, controlling the fingertip mouse to perform a click operation at the corresponding coordinates;

When the action of the fingertip of the target finger is classified into the non-click behavior category, the fingertip mouse is controlled to perform a non-click operation at the corresponding coordinates.
The realization method of tablet fingertip mouse according to claim 7, is characterized in that,

The action of the fingertip of the target finger is classified into the click behavior category or the non-click behavior category, specifically:

Determining the probability that the action of the fingertip of the target finger is classified into the click behavior category, and the probability that it is classified into the non-click behavior category;

The probability that the action of the fingertip of the target finger is classified into the click behavior category, and the probability of being classified into the non-click behavior category are respectively compared with the preset classification probability threshold, and the action of the fingertip of the target finger is Be classified into the click behavior category or the non-click behavior category corresponding to the probability greater than the classification probability threshold.
A fingertip mouse realization system is characterized in that it comprises:

The camera processing module is used to receive the angle adjustment of the image acquisition device of the tablet computer, so that the tablet image acquisition device is aligned with the finger activity area;

An image acquisition module, configured to acquire the picture of the finger activity area at the current moment through the image acquisition device;

A finger acquisition module, configured to acquire the position and action of the fingertip of the target finger in the picture acquired by the image acquisition module;

The finger analysis module is used to determine the position of the fingertip mouse in the display interface of the tablet computer according to the position of the fingertip of the target finger acquired by the finger acquisition module; The motion of the fingertip of the target finger acquired by the module controls the fingertip mouse to perform an operation corresponding to the motion at the determined position.
The flat-panel fingertip mouse realization system according to claim 9, wherein the finger acquisition module is further configured to, before acquiring the position and action of the fingertip of the target finger, according to the preset finger type in the The target finger is determined in the screen of the finger active area.