CN107193384B - Switching method of mouse and keyboard simulation behaviors based on Kinect color image - Google Patents
Switching method of mouse and keyboard simulation behaviors based on Kinect color image Download PDFInfo
- Publication number
- CN107193384B CN107193384B CN201710512501.1A CN201710512501A CN107193384B CN 107193384 B CN107193384 B CN 107193384B CN 201710512501 A CN201710512501 A CN 201710512501A CN 107193384 B CN107193384 B CN 107193384B
- Authority
- CN
- China
- Prior art keywords
- behavior
- color image
- mouse
- switching
- simulation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Processing Or Creating Images (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The switching method based on the Kinect color image in mouse and keyboard simulation behaviors adopts a mouse and keyboard behavior simulation switching system to be respectively connected with a Kinect sensor and output equipment; the switching system comprises five modules, and a main program module is respectively connected with a data capturing module, a mouse behavior simulation module, a keyboard behavior simulation module and a behavior switching module; extracting pixel colors of corresponding skeleton points on a color image in the data capturing module by adopting an OpenCV development library, and respectively putting the color image and skeleton data into different threads; the mouse behavior simulation module is responsible for simulating the clicking behavior of the left button, the double clicking behavior and the clicking behavior of the right button of the mouse; the keyboard behavior simulation module is responsible for simulating the pressing and bouncing behaviors of the keys; the behavior switching module is responsible for switching states, namely switching of mouse behavior simulation and keyboard behavior simulation. The invention has the advantages of applying the color image to the action recognition, reducing the number of actions to a great extent and avoiding the mutual influence among the actions.
Description
Technical Field
The invention belongs to the technical field of human-computer interaction, and mainly aims to apply a color image captured by a Kinect sensor to action recognition to realize behavior simulation switching of a mouse and a keyboard. In order to enable a system for simulating mouse and keyboard behaviors to use fewer action types and have more flexible maneuverability, a new design idea is provided on the basis of the former people, and the behavior simulation switching of the mouse and the keyboard is realized by reading Kinect sensor color image data and the characteristic of self-contained skeleton tracking. By adding color image identification, the number of action types to be identified can be effectively reduced, so that mutual interference among actions is reduced, the selectable range of pixel colors is wide, a larger identification domain is provided, and meanwhile, the bone point is positioned at the central position of the visual field of the sensor by calibrating the color of the bone point, so that the purposes of improving the accuracy and precision of action identification are achieved.
Background
As a crucial link in the field of computer technology research, the human-computer interaction technology attracts more and more students to search for a more convenient and rapid interaction mode[1]. The interactive mode takes the significance of the milestone, and people are skilled in using a mouse and operating a computer through a visual user interface. The defects and limitations of the operation modes of the traditional mouse and keyboard are more and more prominent, and people no longer meet the traditional interaction mode[2]Therefore, people are encouraged to continuously search other more natural and comfortable interaction modes. The following table is a relevant study of motion recognition in terms of human-computer interaction.
TABLE 1 relevant research literature for motion recognition in human-machine interaction
Along with the continuous improvement of human-computer interaction technology, sensors capable of capturing depth information appear, such as Kinect sensors, which have the functions of dynamically capturing human body color images, depth images and bone data, performing voice recognition and voice source direction positioning through microphone input, and the like, Kinect V2 can track 6 sets of complete bones and 25 joints of each person, capture information of each joint, understand human body action behaviors, send operation instructions of a mouse and a keyboard to a computer, and control machine equipment, games and the like depending on the computer by taking the computer as a medium, so that human-computer interaction is not limited in distance, position and hardware, and the operation is more natural, convenient and more accordant with the original intention of human-computer interaction. However, human-computer interaction is realized through human body action recognition (positioning of a human body static position, tracking of a human body motion state, recognition of different human body postures and understanding of different human body behaviors, so that a plurality of problems that interaction among different action types is caused by excessive number of action types, another simulation behavior occurs in one simulation behavior, how to dynamically extract a color image, cursor shake caused by bone point shake and the like can occur.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings of the prior art and provides a switching method for simulating the behaviors of a mouse and a keyboard based on motion recognition and color image recognition.
(1) On the basis of extracting the pixel color at the position of the skeleton point after the conversion on the color image, the Kinect sensor automatically calculates the rotation angle and rotates by detecting whether the calibrated color is red or other colors through calibrating the color, so that the skeleton point is positioned at the center of the visual field of the sensor, and the aim of improving the accuracy and precision of action identification is fulfilled.
(2) In order to reduce the number of action types, the color image captured by the Kinect is applied to action recognition, the action recognition is converted into a range of four channel values for judging pixel colors corresponding to skeleton points on the color image, the switching of keyboard and mouse simulation actions is realized, the number of the action types is reduced to a great extent, the mutual interference among the actions is avoided, and therefore the phenomenon that another simulation action appears in one simulation action is avoided.
The technical scheme of the invention is as follows:
the invention relates to a switching method of mouse and keyboard simulation behaviors based on a Kinect color image, which is characterized in that a switching system for realizing mouse and keyboard behavior simulation is respectively connected with a Kinect sensor and output equipment; the switching system comprises five modules, and is formed by connecting a main program module with a data capturing module, a mouse behavior simulation module, a keyboard behavior simulation module and a behavior switching module respectively; in the system implementation process, each module is implemented according to the principle of high cohesion and low coupling, and the division of labor of each module is clear; the extraction of pixel colors of corresponding bone points on a color image in the data capture module adopts an OpenCV development library, and the color image and the bone data are respectively obtained and put into different threads; the mouse behavior simulation module is responsible for simulating the clicking behavior of the left button, the double clicking behavior and the clicking behavior of the right button of the mouse; the keyboard behavior simulation module is responsible for simulating the pressing and bouncing behaviors of the keys; the behavior switching module is responsible for switching states, namely switching between mouse behavior simulation and keyboard behavior simulation; in order to simplify the action types and facilitate the operation, the number of the action types and the complexity of mutual switching among the actions are reduced as much as possible, the used action types are arranged on a left hand and a right hand, and the switching among different actions is concentrated on a single skeleton point, so that the same functional requirements are realized, the number of the skeletons is reduced as much as possible, and the flexibility of the operation is increased; the switching mode of the behavior switching module is realized by dynamically extracting the pixel color of the mapping position corresponding to the hand left skeleton point on the color image, when the color is white, the current simulation state is keyboard behavior simulation, and when the color is black, the current simulation state is mouse behavior simulation, as shown in table 2;
firstly, capturing color images and skeleton point data by a Kinect sensor, storing color image pixel color data into a Mat variable in a matrix form, dynamically extracting pixel colors at positions of Handleft skeleton points after corresponding mapping in the matrix through an OpenCV (open computer vision library), judging the state of a current simulation behavior according to four channels of the extracted pixel colors, and simultaneously judging the current execution function of the system by the system according to the state of the current simulation behavior and the captured skeleton point position data; the data of the system comprises color image data and skeleton point three-dimensional coordinate data, and the processing process comprises data capturing, pixel extraction, state judgment and action type judgment.
The invention has the advantages that the system can use less action types and has more flexible maneuverability in the process of simulating the switching of the mouse and the keyboard actions, and the number of the action types to be identified is effectively reduced by adding the color pattern identification; meanwhile, the color of the skeleton point is calibrated, so that the skeleton point is positioned at the central position of the visual field of the sensor, and the aims of improving the accuracy and precision of action behavior recognition are fulfilled.
TABLE 2 switching behavior simulation function table
Drawings
FIG. 1 is a schematic block diagram of a system according to the present invention;
FIG. 2 is an overall data flow diagram;
FIG. 3 is a flow chart of a pixel color algorithm;
FIG. 4 is a flow chart of a behavior switch algorithm;
FIG. 5 is a diagram of behavior switching classes;
FIG. 6 is a diagram of a mapping relationship between bone space coordinates and screen coordinates;
FIG. 7 is a graph of the positional relationship of the captured bone points and the Kinect sensor;
fig. 8 is a schematic diagram of a motion trajectory of a cursor under different pixel colors.
Detailed Description
Referring to fig. 1 and 2, a switching method for simulating behaviors of a mouse and a keyboard based on a Kinect color image is characterized in that a switching system for realizing the mouse and keyboard behavior simulation is respectively connected with a Kinect sensor and output equipment; the switching system comprises five modules, and is formed by connecting a main program module with a data capturing module, a mouse behavior simulation module, a keyboard behavior simulation module and a behavior switching module respectively; in the system implementation process, each module is implemented according to the principle of high cohesion and low coupling, and the division of labor of each module is clear; the extraction of pixel colors of corresponding bone points on a color image in the data capture module adopts an OpenCV development library, and the color image and the bone data are respectively obtained and put into different threads; the mouse behavior simulation module is responsible for simulating the clicking behavior of the left button, the double clicking behavior and the clicking behavior of the right button of the mouse; the keyboard behavior simulation module is responsible for simulating the pressing and bouncing behaviors of the keys; the behavior switching module is responsible for switching states, namely switching between mouse behavior simulation and keyboard behavior simulation; in order to simplify the action types and facilitate the operation, the number of the action types and the complexity of mutual switching among the actions are reduced as much as possible, the used action types are arranged on a left hand and a right hand, and the switching among different actions is concentrated on a single skeleton point, so that the same functional requirements are realized, the number of the skeletons is reduced as much as possible, and the flexibility of the operation is increased; the switching mode of the behavior switching module is realized by dynamically extracting the pixel color of the mapping position corresponding to the hand left skeleton point on the color image, when the color is white, the current simulation state is keyboard behavior simulation, and when the color is black, the current simulation state is mouse behavior simulation, as shown in table 2;
firstly, capturing color images and skeleton point data by a Kinect sensor, storing color image pixel color data into a Mat variable in a matrix form, dynamically extracting pixel colors at positions of Handleft skeleton points after corresponding mapping in the matrix through an OpenCV (open computer vision library), judging the state of a current simulation behavior according to four channels of the extracted pixel colors, and simultaneously judging the current execution function of the system by the system according to the state of the current simulation behavior and the captured skeleton point position data; the data of the system comprises color image data and skeleton point three-dimensional coordinate data, and the processing process comprises data capturing, pixel extraction, state judgment and action type judgment.
The data flow of the system and the overall flow thereof are shown in fig. 2, wherein the parallelogram part represents the main data related to the system, the rectangle represents the main processing procedure, and the diamond represents the decision procedure.
The specific implementation mode of the invention is as follows:
a first part:
the behavior switching module mainly comprises human skeleton and color image matching and pixel color extraction, wherein a coordinate conversion technology is involved in the human skeleton and color image matching, the pixel color extraction needs to dynamically extract the pixel color of a corresponding position on a color image by means of an OpenCV (open content description language) graphic library, the values of the four channels of r, g, b and a are calculated, whether the numerical values of the four channels meet the threshold value requirement is judged, and the switching of the relevant state is realized.
The human skeleton captured by the Kinect V2 sensor and the color image coordinate system do not belong to the same dimension, the human skeleton captured by the Kinect is registered with the color image through coordinate conversion, the pixel color on the color image is extracted through the converted two-dimensional coordinates of the Handleft skeleton point, and the algorithm flow chart is shown in fig. 3.4.
The dimensions of the coordinate systems of the human skeleton and the screen captured by the Kinect V2 sensor are different, and the mapping of the hand skeleton point and the screen coordinate is realized through coordinate mapping[51]And by adjusting lx,lyThe value controls the degree of influence of the moving distance of the cursor in the horizontal direction and the vertical direction in unit time by the moving distance of the bone point, the switching of the simulation behavior is controlled by changing the extracted pixel color of the Handleft bone point in the color image, and the flow chart of the algorithm is shown in FIG. 3.
A second part:
(1) behavior switching programming
The class diagram of behavior switching is shown in fig. 5, where a StateSwitch class inherits from a baseteswitch class, the baseteswitch class mainly takes charge of converting a three-dimensional coordinate position of a skeletal point to a two-dimensional coordinate position on a color image with an upper left corner as an origin and performing state switching, an Execute function takes charge of implementing a subclass function, a Vector2 class describes a coordinate point of any point on the color image, and the StateSwitch class mainly takes charge of specifically implementing behavior switching.
(2) Description of algorithms
The behavior switching module relates to the mutual switching of mouse behavior simulation and keyboard behavior simulation, the switching is realized by dynamic extraction of pixel colors on a color image, because the positions of skeleton points captured by a Kinect sensor and the positions of human body skeleton points on the color image need to be paired, one-to-one correspondence on the positions is realized, a coordinate conversion method is adopted in the text, the pairing of coordinates is realized by affine transformation such as movement, scaling, rotation and the like, after the Kinect coordinates are paired, each pixel color value on the color image is stored in a matrix, the subscript of the corresponding matrix is found by the coordinates of the HandLeft skeleton points after conversion, so that the pixel color value at the position of the HandLeft skeleton points on the color image is taken out, four channels of the pixel color are extracted, if the values of the four channels meet white color, the switching is carried out to the keyboard behavior simulation, if the values of the four channels meet black color, the mouse behavior simulation is switched to, and the number of the action types can be reduced to a great extent by a method of replacing part of action behavior recognition by dynamic recognition of color image pixel colors, so that mutual influence among the action behaviors is avoided.
Because the position of the skeleton point is far away from the Kinect sensor or at the edge of the visual field range of the sensor, the captured skeleton point information is wrong, the Kinect sensor automatically calculates the rotation angle theta and rotates to the direction facing the skeleton point by calibrating the color of the skeleton point to be red or other colors on the basis of pixel color extraction when detecting that the color of the skeleton point position after conversion on a color image is the calibration color, and the direction can enable the skeleton point to be positioned at the center position of the visual field of the sensor, so that the wrong skeleton point information is avoided being obtained, the purposes of improving the accuracy rate and the identification precision of action identification are achieved, and the related specific algorithm is implemented as follows.
(3) Coordinate mapping
In the process of simulating the movement of the cursor, the cursor is required to move along with the movement of the hand right skeleton point and has the same movement track, and the dimensions of the skeleton point captured by the Kinect sensor and the coordinate system of the screen are different, as shown in fig. 7, the hand right skeleton point of the human body takes the sensor as the origin (O)1) The point existing on the display screen is with its upper left corner as the origin (O)2) Two-dimensional coordinate, with X-axis direction horizontally to the right and Y-axis direction vertically downward[50]Therefore, it is required to add (O)1) As the origin hand left bone point coordinates (X, Y, Z) (-l)x＜＝x＜＝lx,-ly＜＝y＜＝ly) Is mapped to2) Screen coordinates (x, y, z) which are the origin of coordinates.
The coordinate mapping relation between the human hand Right skeleton point and the screen is shown as a formula 3-1, 3-2, 3-3, 3-4, wherein the coefficient l in the formulax,lyThe method is used for controlling the degree of influence of the moving distance of the cursor in the horizontal direction and the vertical direction on the moving distance of the bone point in unit time.
In the above formula, screen width is the width of the screen, screen height is the height of the screen, (X, Y, Z) is the three-dimensional coordinate of the left-hand skeleton point with the KinectV2 sensor as the origin, and (X, Y, Z) is the coordinate on the screen after mapping.
(4) Coordinate transformation
The skeleton coordinates captured by the Kinect sensor and the coordinates of the pixel colors on the color image are not in the same coordinate system, the dimensions of the coordinate systems are different, and meanwhile, the requirement that the positions of the dynamically extracted pixel colors on the color image and the positions of the human skeleton points captured by the Kinect are in one-to-one correspondence is met, so that the positions of the skeleton points and the positions of the human skeleton on the color image need to be matched.
The specific steps need to perform scaled affine transformation on the coordinate points of the skeleton points, so as to convert the coordinate points into a three-dimensional coordinate system of a depth image, and perform affine transformation of rotation and translation, so as to convert the coordinate points into a two-dimensional coordinate system of a color image, so that a projection calculation formula can be adopted for calculation, and the mapping relationship is as shown in formula 5-1:
in the above formula, an S scaling matrix, R is a rotation matrix, T is a translation matrix, (X, Y, Z) are coordinates of the skeleton points in the color image after being converted, and (X, Y, Z) are three-dimensional coordinates of the skeleton points with the Kinect sensor as an origin. The point coordinates of the corresponding color image are shown in equation 5-2:
wherein (f)x,fy) Is a scale parameter on the color image captured by the sensor, (o)x,oy) The center point on the color image captured by the sensor. The fitting results are shown in the formula 5-3
Through affine transformation such as rotation, scaling and translation of the positions of the skeleton points captured by the Kinect sensor, the skeleton points in three-dimensional coordinates can be displayed on a color image in two-dimensional coordinates, and matching between the positions of the skeleton points captured by the Kinect sensor and the positions of the skeleton points of the human body on the color image can be realized, namely, the positions on the color image correspond to one another one by one.
(5) Pixel color extraction
The behavior switching module relates to switching between keyboard behavior simulation and mouse behavior simulation, if behavior simulation is still switched by adopting motion recognition, the number of motion behavior recognition is excessive, the pixel color at the position of a skeleton point after color image up-conversion is extracted, and the switching of the simulation behaviors is realized according to the value ranges of four channels of the pixel color.
First, color image data captured by Kinect V2 is acquired by an OpenCV graphics library function, and a color image is displayed by reading a value of each captured pixel color, and since directions of an X axis and a Y axis on the color image and a matrix index position are diametrically opposite, a pixel color of an (X, Y) coordinate point on the color image is obtained by an (Y, X) index of a matrix storing the color image data.
Copying pixel color data of the extracted color image into a Mat matrix of an OpenCV (open CV) graphic library, extracting pixel colors through a Mat matrix subscript, and firstly, matching the skeleton point position with the color image through coordinate conversion to enable the position after the skeleton point conversion of the Handleft to be the position for extracting the pixel colors, wherein the calculation formulas of four channels of the pixel colors are shown as formulas 5-4, 5-5, 5-6 and 5-7.
b＝I.at<Vec4b>(y,x).val[0](5-4)
g＝I.at<Vec4b>(y,x).val[1](5-5)
r＝I.at<Vec4b>(y,x).val[2](5-6)
alpha＝I.at<Vec4b>(y,x).val[3](5-7)
In the above formula, I stores a matrix of color data, b, r, g, a are four channels of pixel colors, and (y, x) coordinates are positions where pixel color points are extracted.
(6) Effect of Pixel color on motion recognition accuracy
The information of the bone points captured by the Kinect sensor is influenced by the relative position of the human body and the sensor, when the human body is far away from the camera or at the edge of the visual field of the camera, the captured information of the bone is often wrong, and the position of the bone point in the center of the visual field of the Kinect sensor is the best position for capturing the information of the bone point. The color image resolution ratio that Kinect inductor caught is higher, and the field of vision is wider, can not influenced by human body and inductor relative position, and Kinect inductor provides the function of adjusting the camera orientation, draw the pixel colour of different skeleton points through OpenCV in this paper, when the pixel colour of drawing is red or other colours, and when skeleton point takes place the displacement, Kinect inductor's camera can be automatic through rotating this skeleton point of orientation from top to bottom, thereby make this skeleton point be located the central point in the inductor field of vision, guaranteed that the information of this skeleton point of catching is correct from this, with this purpose that has reached the rate of accuracy and the precision that has improved this action discernment.
As shown in fig. 7, when a bone point to be captured is above the Kinect sensor and an angle with the xy plane is θ, an angle a of the camera is set to | θ |, a bone point to be captured is below the Kinect sensor and an angle with the xy plane is θ, and an angle a of the camera is set to-1 | θ |.
Assuming that the coordinate of the bone point A to be captured is (x, y, z), and the angle theta is to be calculated, firstly, the point A is projected to an OYZ plane, the coordinate of the point B after projection is obtained to be (o, y, z), and the included angle between the point B and the OXZ plane is calculated to be the required included angle theta, and the formula is shown as formulas 5 to 8.
Through setting the rotating angle theta of the inductor, the bone point is positioned at the center of the visual field of the inductor, so that the phenomenon that the captured bone point information is wrong due to the fact that the bone point is at the edge of the inductor is avoided, and the accuracy and precision of action behavior recognition are improved.
And a third part:
(1) pixel color extraction experiment
The method comprises the steps that a behavior switching module needs to switch a keyboard simulation behavior and a mouse simulation behavior, if the behavior switching module adopts the behavior recognition to switch, the number of the motions is increased to a great extent, therefore, the switching is carried out by judging the pixel color of the position, corresponding to the converted bone point, of a color image, wherein the extraction of the pixel color of the converted bone point is involved, an extraction method is provided, the feasibility of a pixel color extraction mode is verified through experiments, and the specific experiment method is as follows.
In the pixel color extraction experiment, a Neck bone point of 25 bone points provided by a KinectV2 sensor is selected, coordinates obtained by converting the Neck bone point into a color image are assumed to be (x, y), a pixel color of the coordinate point (x, y) in the color image is extracted, and a circle with the pixel color as an origin and the radius R being 15 is drawn by using the pixel color.
The pixel color of the Neck skeleton point with coordinates (x, y) on the color image after conversion is changed into white and red, a program draws a white circle and a red circle respectively according to the pixel color extracted at the (x, y) point, the color of a circular frame displayed on the color image is the same as the pixel color extracted from the Neck skeleton point, namely the position of the skeleton point captured by the Kinect sensor and the position of human skeleton on the color image can be matched through coordinate conversion, the skeleton point moves along with the movement of the human body, and a coordinate conversion formula is suitable for the movement of the human body, so that the dynamic extraction of the pixel color is realized.
(2) Experiment of influence of pixel color on cursor movement
The chapter provides a pixel color extraction mode, and experiments verify the feasibility of the mode, however, whether the extracted pixel color can be applied to actual development needs to be further verified through experiments. The specific implementation method is as follows.
First, before the experiment, the uncombined color image recognition and the combined color image recognition were compared, and data of the motion type, the number of the motion types, the type of the bone points, and the number of the bone points in the two states are given as shown in table 3.
TABLE 3 comparison under different State conditions
It is concluded from table 3 that after using color image recognition, the number of motion types and bone points is significantly reduced, thereby also avoiding the interaction between multiple motions, and as the functions are increasing, the number of motion behaviors is more enormous, and after adding color image recognition, because the recognizable color range is wider, the number of motion types can be reduced to a great extent, i.e. for systems with more functional requirements, color image recognition based on pixel color dynamic extraction is more applicable.
In this experiment, the hand right skeleton point is moved and kept in motion all the time, and the coordinates of the cursor on the screen are obtained at intervals of t ═ 1s as (x, y) as shown in table 4. On the basis of the two-dimensional coordinates of the screen, a z-axis representing the sampling time is added to draw a three-dimensional coordinate axis as shown in fig. 8, where the (e, d, c, b, a) position in the figure is a point where the extracted pixel color is black, and the other positions are points where the extracted pixel color is white.
TABLE 4 extraction of Cursor screen coordinates for different pixel colors per unit time
As can be seen from fig. 8, as the time of the Z axis increases, the cursor moves for a period of time, stops moving, and then starts moving again, where the coordinate (x, y) of the cursor at the position (e, d, c, b, a) does not change, that is, the movement is stopped, and the color of the pixel extracted at the position of the changed HandLeft bone point on the color image is black, and at other positions, the cursor moves along with the movement of the HandLeft bone point all the time, and the color of the pixel extracted at the position of the changed HandLeft bone point on the color image is white.
(3) Experiment of influence of pixel color on identification precision
The invention provides a method for capturing skeleton points, which comprises the steps that when the skeleton points to be captured are far away from a Kinect sensor or are at the edge of the visual field of the sensor, the obtained skeleton point information is wrong, the color of a pixel at the position of the skeleton point after conversion on a calibrated color image is red, and when the skeleton point is displaced, a camera of the Kinect sensor automatically faces the skeleton point marked as red towards a human body through calculating the angle theta value, so that the skeleton point is located at the center of the visual field of the sensor, the captured skeleton point information is correct, and the purpose of improving the action recognition accuracy and precision is achieved. The experimental methods are given herein, and the specific methods are as follows.
In the experiment, the click of the right button of the mouse is simulated by the fist of the right hand, and if the right hand punches in each simulationThe mouse right click behavior simulation method based on the Kinect comprises the steps that a click of a right mouse button is not triggered, an error event is considered, the right hand is located at the edge of a Kinect visual field, colors of HandRight skeleton points are calibrated to be red and white respectively, when the colors are red, a Kinect sensor can calculate a theta value, the direction of the HandRight skeleton points is achieved by rotating the theta angle, when the colors are white, a camera keeps a normal direction and does not rotate, a simulation experiment of mouse right click behaviors for 10 times is conducted in the experiment, and the total error generation times S corresponding to simulation times i are recordediAs in table 5.
TABLE 5 Total number of errors due to simulation of behavior for different pixel colors
With the increasing of the times of right click simulation behaviors of the mouse, when the color of the extracted pixel is red, the total times of errors generated by the right click simulation behaviors of the mouse are obviously lower than the condition that the color of the extracted pixel is white, namely, whether the Kinect sensor automatically calculates the rotation angle theta is judged by calibrating the color of the pixel at the skeleton point is verified, the skeleton point is positioned at the center of the visual field of the sensor through the rotation angle theta, and therefore the method for improving the accuracy and precision of action identification is feasible.
The fourth part:
the system simulates cursor movement of mouse behavior, clicking of a left key, double clicking, clicking of a right key, dragging of a window and pressing and bouncing of keys of keyboard behavior, and at this stage, the simulation behavior related to the system is tested, and the specific test process is as follows.
(1) And (3) performing a mouse right-key simulation behavior function test, wherein a tester firstly makes actions of naturally drooping a left hand and making a fist by a right hand, and observes whether a mouse right-key clicking behavior appears on a screen or not, and the mouse right-key clicking behavior appears along with the action of making the fist by the right hand.
(2) The method comprises the following steps of performing keyboard key pressing and bouncing behavior function test, moving a cursor through a right palm and clicking a keyboard opening button on a system interface, then moving the cursor to a key of a virtual keyboard, performing fist making and palm stretching actions, observing whether a key pressing behavior occurs, pressing the a key on the virtual keyboard after a left hand makes a fist, and bouncing the pressed a key when the left hand is in an extending state.
(3) And (4) performing cursor movement behavior function test, firstly moving the position of the right palm, and observing that the movement track of the cursor is the same as that of the right palm.
Through testing, the tested system function can be normally realized.
Reference documents:
[1] the Wanyankai, dynamic gesture recognition and application research [ D ] Xian electronics technology university, 2014.
[2]S.Szilvia,M.Zichar,F.Attila.Gesture-based computer mouse usingKinect sensor[c].In Proceeding ofthe 5th IEEE International Conference onCognitive Infocommunications.IEEE,2014:419-424.
[3]Tran N X,Phan H,Dinh V V,et al.Wireless Data Glove for Gesture-Based Robotic Control[J].Human-Computer Interaction.Novel Interaction Methodsand Techniques,2009,5611:271-280.
[4] Liberdong. model-based human movement analysis [ D ]. university of hiking, 2012.
[5] Anyseaman, wish you far new, you xu, etc. spatio-temporal appearance modeling and recognition of continuous dynamic gestures [ J ] computer science, 2000,23(8): 824-.
[6]Michal W F.Orientation Histograms for Hand Gesture Recognition[J].Mitsubishi Electric Research Labs,1999:296--301.
[7]S.Qin,X.Zhu,Y.Yang,et a.Real-time hand gesture recognition fromdepth images using convex shape decomposition method[J].Journal of SignalProcessing Systems,2014:47–58.
[8]A.Michael,Y.Yang,L.Li.Joint Filtering:Enhancing gesture andmousemovement in Microsoft Kinect application[C].In Proceedings of the 12thInternational Conference on Fuzzy Systems and Knowledge Discovery.IEEE,2015:2528–2532
[9] A skin color detection algorithm under variable illumination [ J ] computer engineering and application, 2002,38(24): 123-.
Claims (1)
1. A switching method for simulating behaviors of a mouse and a keyboard based on a Kinect color image is characterized in that a switching system for realizing the behavior simulation of the mouse and the keyboard is respectively connected with a Kinect sensor and output equipment; the switching system comprises five modules, and is formed by connecting a main program module with a data capturing module, a mouse behavior simulation module, a keyboard behavior simulation module and a behavior switching module respectively; the extraction of pixel colors of corresponding bone points on a color image in the data capture module adopts an OpenCV development library, and the color image and the bone data are respectively obtained and put into different threads; the mouse behavior simulation module is responsible for simulating the clicking behavior of the left button, the double clicking behavior and the clicking behavior of the right button of the mouse; the keyboard behavior simulation module is responsible for simulating the pressing and bouncing behaviors of the keys; the behavior switching module is responsible for switching states, namely switching between mouse behavior simulation and keyboard behavior simulation; the used action types are arranged on the left hand and the right hand, and the switching between different actions is centralized on a single skeleton point, so that the same function requirement is realized, the number of skeletons is reduced as much as possible, and the operation flexibility is increased; the switching mode of the behavior switching module is realized by dynamically extracting the pixel color of the mapping position corresponding to the hand left skeleton point on the color image, when the color is white, the current simulation state is keyboard behavior simulation, and when the color is black, the current simulation state is mouse behavior simulation;
firstly, capturing color images and skeleton point data by a Kinect sensor, storing color image pixel color data into a Mat variable in a matrix form, dynamically extracting pixel colors at positions of Handleft skeleton points after corresponding mapping in the matrix through an OpenCV (open computer vision library), judging the state of a current simulation behavior according to four channels of the extracted pixel colors, and simultaneously judging the current execution function of the system by the system according to the state of the current simulation behavior and the captured skeleton point position data; the data of the system comprises color image data and skeleton point three-dimensional coordinate data, and the processing process comprises data capturing, pixel extraction, state judgment and action type judgment;
wherein, the corresponding mapping relation in the matrix is shown as a formula 1-1:
in the formula, S is a scaling matrix, R is a rotation matrix, T is a translation matrix, (X, Y and Z) are coordinates of the skeleton points in the color image after the skeleton points are converted, and (X, Y and Z) are three-dimensional coordinates of the skeleton points with a Kinect sensor as an origin; the point coordinates of the corresponding color image are shown in equations 1-2:
wherein (f)x,fy) Is a scale parameter on the color image captured by the sensor, (o)x,oy) The center point on the color image captured by the sensor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710512501.1A CN107193384B (en) | 2017-06-29 | 2017-06-29 | Switching method of mouse and keyboard simulation behaviors based on Kinect color image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710512501.1A CN107193384B (en) | 2017-06-29 | 2017-06-29 | Switching method of mouse and keyboard simulation behaviors based on Kinect color image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107193384A CN107193384A (en) | 2017-09-22 |
| CN107193384B true CN107193384B (en) | 2020-01-10 |
Family
ID=59881514
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710512501.1A Expired - Fee Related CN107193384B (en) | 2017-06-29 | 2017-06-29 | Switching method of mouse and keyboard simulation behaviors based on Kinect color image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107193384B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108334189A (en) * | 2017-09-26 | 2018-07-27 | 郑州升达经贸管理学院 | A kind of method and apparatus that human-computer interaction is realized based on camera |
| CN108153421B (en) * | 2017-12-25 | 2021-10-01 | 深圳Tcl新技术有限公司 | Somatosensory interaction method and device and computer-readable storage medium |
| CN112558850B (en) * | 2020-12-16 | 2022-04-26 | 深圳市巨烽显示科技有限公司 | Multi-display mouse seamless moving method and device, computer equipment and medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103713741A (en) * | 2014-01-08 | 2014-04-09 | 北京航空航天大学 | Method for controlling display wall through gestures on basis of Kinect |
| CN104589356A (en) * | 2014-11-27 | 2015-05-06 | 北京工业大学 | Dexterous hand teleoperation control method based on Kinect human hand motion capturing |
| CN104850230A (en) * | 2015-05-26 | 2015-08-19 | 福州大学 | Brain-computer interface control method for simulating keyboard and mouse |
| CN105469043A (en) * | 2015-11-20 | 2016-04-06 | 苏州铭冠软件科技有限公司 | Gesture recognition system |
| CN106055091A (en) * | 2016-05-16 | 2016-10-26 | 电子科技大学 | Hand posture estimation method based on depth information and calibration method |
| KR20170010288A (en) * | 2015-07-18 | 2017-01-26 | 주식회사 나무가 | Multi kinect based seamless gesture recognition method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9696813B2 (en) * | 2015-05-27 | 2017-07-04 | Hsien-Hsiang Chiu | Gesture interface robot |
-
2017
- 2017-06-29 CN CN201710512501.1A patent/CN107193384B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103713741A (en) * | 2014-01-08 | 2014-04-09 | 北京航空航天大学 | Method for controlling display wall through gestures on basis of Kinect |
| CN104589356A (en) * | 2014-11-27 | 2015-05-06 | 北京工业大学 | Dexterous hand teleoperation control method based on Kinect human hand motion capturing |
| CN104850230A (en) * | 2015-05-26 | 2015-08-19 | 福州大学 | Brain-computer interface control method for simulating keyboard and mouse |
| KR20170010288A (en) * | 2015-07-18 | 2017-01-26 | 주식회사 나무가 | Multi kinect based seamless gesture recognition method |
| CN105469043A (en) * | 2015-11-20 | 2016-04-06 | 苏州铭冠软件科技有限公司 | Gesture recognition system |
| CN106055091A (en) * | 2016-05-16 | 2016-10-26 | 电子科技大学 | Hand posture estimation method based on depth information and calibration method |
Non-Patent Citations (2)
| Title |
|---|
| "基于Kinect的体感手势识别系统的研究";沈世红;《第八届和谐人机环境联合学术会议》;20121123;全文 * |
| "基于Kinect骨骼跟踪功能实现PC的手势控制";王明东;《漳州职业技术学院学报》;20120630;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107193384A (en) | 2017-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Rautaray | Real time hand gesture recognition system for dynamic applications | |
| CN106598227B (en) | Gesture identification method based on Leap Motion and Kinect | |
| Shibly et al. | Design and development of hand gesture based virtual mouse | |
| US6204852B1 (en) | Video hand image three-dimensional computer interface | |
| Banerjee et al. | Mouse control using a web camera based on colour detection | |
| Jing et al. | Human-computer interaction using pointing gesture based on an adaptive virtual touch screen | |
| CN110515455B (en) | Virtual assembly method based on Leap Motion and cooperation in local area network | |
| CN107357428A (en) | Man-machine interaction method and device based on gesture identification, system | |
| CN109145802B (en) | Kinect-based multi-person gesture man-machine interaction method and device | |
| CN107193384B (en) | Switching method of mouse and keyboard simulation behaviors based on Kinect color image | |
| CN106774938A (en) | Man-machine interaction integrating device based on somatosensory device | |
| Liang et al. | Bare-hand depth perception used in augmented reality assembly supporting | |
| O'Hagan et al. | Visual gesture interfaces for virtual environments | |
| Boruah et al. | Development of a learning-aid tool using hand gesture based human computer interaction system | |
| Hartanto et al. | Real time hand gesture movements tracking and recognizing system | |
| Aggarwal et al. | An Approach to Control the PC with Hand Gesture Recognition using Computer Vision Technique | |
| Titlee et al. | A novel design of an intangible hand gesture controlled computer mouse using vision based image processing | |
| Conci et al. | Natural human-machine interface using an interactive virtual blackboard | |
| CN105929946B (en) | A kind of natural interactive method based on virtual interface | |
| KR20160141023A (en) | The method of dynamic and static gesture recognition using depth camera and interface of immersive media contents | |
| KR100491432B1 (en) | Interactive exhibition system using multi-modal user interface | |
| Gope et al. | Interaction with Large Screen Display using Fingertip & Virtual Touch Screen | |
| Agarwal et al. | Mp-feg: media player controlled by facial expressions and gestures | |
| Khaliq et al. | Virtual Mouse Implementation Using Color Pointer Detection | |
| Ansari et al. | Air Pose Canvas with Hand Pose Estimation using Machine Learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20200110 Termination date: 20200629 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |