CN107967061A - Man-machine interaction method and device - Google Patents
Man-machine interaction method and device Download PDFInfo
- Publication number
- CN107967061A CN107967061A CN201711391520.XA CN201711391520A CN107967061A CN 107967061 A CN107967061 A CN 107967061A CN 201711391520 A CN201711391520 A CN 201711391520A CN 107967061 A CN107967061 A CN 107967061A
- Authority
- CN
- China
- Prior art keywords
- key point
- gestures
- axis
- palm
- palm key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Human Computer Interaction (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The application provides a kind of man-machine interaction method and device, the described method includes：One group of continuous images of gestures is gathered in the range of pre-set space, and extracts the palm key point of the images of gestures；The gesture model data of preset kind are determined based on the palm key point；Gesture model data template data corresponding with the preset kind is matched；The corresponding touch action of template data obtained according to matching performs interactive operation, so as to fulfill the human-computer interaction of fine gesture motion.And the application realizes the interactive operation of user and terminal device by extracting the palm key point of images of gestures, directly contacted with terminal device without limiting user, improve human-computer interaction efficiency.
Description
Technical field
This application involves interaction technique field, more particularly to a kind of man-machine interaction method and device.
Background technology
At present, it is typically that finger touches or cursor key is made an inventory the interactive mode hit in intelligent terminal interactive mode, but
These interactive modes are a kind of specific contact man-machine interaction mode, are not natural human-machine interaction mode.
In the related art, realized by using AR (Augmented Reality, augmented reality) technology contactless
Human-computer interaction, specifically, the intelligent terminal by supporting AR technologies catch picture, and by identifying the human skeleton in picture
(or arm) is acted with the simple significantly action such as arm action to interact.But current AR technologies can only realize letter
Single interaction significantly acted, can not but realize the interaction of fine gesture motion.
The content of the invention
In view of this, the application provides a kind of man-machine interaction method and device, and hand can not be realized to solve existing AR technologies
The problem of interaction of gesture fine movement.
According to the first aspect of the embodiment of the present application, there is provided a kind of man-machine interaction method, the described method includes：
One group of continuous images of gestures is gathered in the range of pre-set space, and the palm for extracting the images of gestures is crucial
Point；
The gesture model data of preset kind are determined based on the palm key point；
Gesture model data template data corresponding with the preset kind is matched；
The corresponding touch action of template data obtained according to matching performs interactive operation.
According to the second aspect of the embodiment of the present application, there is provided a kind of human-computer interaction device, described device include：
Extraction module is gathered, for gathering one group of continuous images of gestures in the range of pre-set space, and extracts the hand
The palm key point of gesture image；
Determining module, for determining the gesture model data of preset kind based on the palm key point；
Matching module, for gesture model data template data corresponding with the preset kind to be matched；
Interactive module is performed, the corresponding touch action of the template data for being obtained according to matching performs interactive operation.
Using the embodiment of the present application, after terminal device collects one group of continuous images of gestures in the range of pre-set space,
The palm key point of images of gestures is extracted, and the gesture model data of preset kind are determined based on palm key point, then by hand
Potential model data template data corresponding with the preset kind is matched, and the template data finally obtained according to matching is corresponding
Touch action performs interactive operation, so as to fulfill the human-computer interaction of fine gesture motion.And the application is by extracting gesture figure
The palm key point of picture realizes the interactive operation of user and terminal device, is directly contacted with terminal device without limiting user,
Improve human-computer interaction efficiency.
Brief description of the drawings
Figure 1A is a kind of embodiment flow chart of man-machine interaction method of the application according to an exemplary embodiment；
Figure 1B is a kind of palm key point schematic diagram of the application according to Figure 1A illustrated embodiments；
Fig. 2A is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment；
Fig. 2 B are a kind of space coordinates schematic diagram of the application according to Fig. 2A illustrated embodiments；
Fig. 3 is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment；
Fig. 4 is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment；
Fig. 5 is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment；
Fig. 6 is a kind of hardware structure diagram of intelligent terminal of the application according to an exemplary embodiment；
Fig. 7 is a kind of example structure figure of human-computer interaction device of the application according to an exemplary embodiment.
Embodiment
Here exemplary embodiment will be illustrated in detail, its example is illustrated in the accompanying drawings.Following description is related to
During attached drawing, unless otherwise indicated, the same numbers in different attached drawings represent the same or similar key element.Following exemplary embodiment
Described in embodiment do not represent all embodiments consistent with the application.On the contrary, they be only with it is such as appended
The example of the consistent apparatus and method of some aspects be described in detail in claims, the application.
It is only merely for the purpose of description specific embodiment in term used in this application, and is not intended to be limiting the application.
It is also intended in the application and " one kind " of singulative used in the attached claims, " described " and "the" including majority
Form, unless context clearly shows that other implications.It is also understood that term "and/or" used herein refers to and wraps
Containing the associated list items purpose of one or more, any or all may be combined.
It will be appreciated that though various information, but this may be described using term first, second, third, etc. in the application
A little information should not necessarily be limited by these terms.These terms are only used for same type of information being distinguished from each other out.For example, do not departing from
In the case of the application scope, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as
One information.Depending on linguistic context, word as used in this " if " can be construed to " ... when " or " when ...
When " or " in response to determining ".
Figure 1A is a kind of embodiment flow chart of man-machine interaction method of the application according to an exemplary embodiment；Figure
1B is a kind of palm key point schematic diagram of the application according to Figure 1A illustrated embodiments, which can be applied to support
The terminal device (such as mobile phone, tablet computer, personal computer etc.) of AR interaction techniques, namely it is provided with energy on the terminal device
The camera device of enough sampling depth images.As shown in Figure 1A, which includes the following steps：
Step 101：One group of continuous images of gestures is gathered in the range of pre-set space, and the palm for extracting images of gestures closes
Key point.
In one embodiment, for terminal device when having detected that user gesture is in the range of pre-set space, starting AR should
With or camera device gather one group of continuous images of gestures (video flowing obtained in such as image acquisition process), one group is continuous
Images of gestures includes multiframe consecutive image, and the frame number of the images of gestures gathered every time is identical.The images of gestures of collection can be
Depth image, each of which pixel are depth of field data, represent distance of the subject away from imaging device.
Wherein, pre-set space scope can be the corresponding spatial dimension of terminal device display screen.
In another embodiment, terminal device can extract the palm key point in every frame images of gestures.Art technology
Personnel it is understood that the extraction of palm key point can be realized by correlation technique, such as deep learning model or with
Machine forest algorithm can realize the extraction of palm key point.The palm key point can be palm artis or except palm close
Other points or both combinations outside section.The number of palm key point can be set according to practical experience, such as Figure 1B
Shown, palm key point includes 21 points, other points by palm artis and in addition to palm joint form.Wherein, hand
Metacarpal joint point has 15 white points；Other points in addition to palm joint have 4 Grey Points and 1 black color dots, black color dots respectively
What is represented is the centre of the palm.
Step 102：The gesture model data of preset kind are determined based on palm key point.
How based on palm key point to determine that the description of the gesture model data of preset kind may refer to following Fig. 2A, figure
The description of any illustrated embodiments of 3- Fig. 5, is not described in detail first herein.
Step 103：Gesture model data template data corresponding with the preset kind is matched.
In one embodiment, gesture model data template data corresponding with preset kind can be carried out phase by terminal device
Compared like degree, and one group of template data is determined based on similarity comparison result, the template data obtained as matching.
Wherein, terminal device can set a similarity threshold according to practical experience, be selected from similarity comparison result
A maximum similarity is selected, and judges whether the maximum similarity exceedes similarity threshold, if, it is determined that the maximum similarity
Corresponding template data, and the template data obtained as matching.
In another embodiment, if the maximum similarity is not less than similarity threshold, then it represents that gesture identification fails, and carries
Show that user inputs gesture again.
Step 104：The corresponding touch action of template data obtained according to matching performs interactive operation.
In one embodiment, terminal device can pre-set template data and the correspondence of touch action, the touch
Action can include click, long-press, slip, the fine gesture motion such as gently sweep, rotates, shrinking, stretching, for original according to user
Custom, corresponding interactive operation is performed to terminal device.For example, the interactive operation that " click " performs is the original touch of user
Clicking operation.So as to which the touch action obtained by images of gestures retains original user's custom so that user is using AR gestures
In interaction too many energy need not be spent to think gesture motion.
In another embodiment, can be with for the pre-set each corresponding template data of touch action of terminal device
The characteristics of protruding the touch action well.
In the present embodiment, after terminal device collects one group of continuous images of gestures in the range of pre-set space, hand is extracted
The palm key point of gesture image, and determine based on palm key point the gesture model data of preset kind, then by gesture model
Data template data corresponding with the preset kind is matched, and corresponding touch of the template data finally obtained according to matching is moved
Make execution interactive operation, so as to fulfill the human-computer interaction of fine gesture motion.And the application is by extracting the hand of images of gestures
Palm key point realizes the interactive operation of user and terminal device, directly contacts, improves with terminal device without limiting user
Human-computer interaction efficiency.
Fig. 2A is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment；
Fig. 2 B are a kind of space coordinates schematic diagram of the application according to Fig. 2A illustrated embodiments, and the present embodiment is real using the application
The above method of example offer is applied, how to determine that the gesture model data instance of preset kind carries out example based on palm key point
Property explanation, as shown in Figure 2 A, include the following steps：
Step 201：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis.
In one embodiment, which is the relative termination equipment coordinate reference system of itself, i.e. terminal device
Display screen is in the plane that X-axis is formed with Y-axis, as shown in Figure 2 B.
Step 202：Adjacent gesture is determined according to coordinate of the palm key point of every frame images of gestures in space coordinates
First variable of palm key point between image, first variable between adjacent images of gestures palm key point along Z axis side
To displacement distance.
In one embodiment, it is with a finger touch display screen to be operated by original user's point touching, then lifts hand
Refer to, therefore, corresponding AR gesture operations can be along Z-direction stretch out a finger, and along Z-direction carry out before and after once or
It is repeatedly mobile, i.e., carry out one or many movements perpendicular to X-axis and the plane of Y-axis composition (i.e. plane where display screen).In addition,
It is with a finger touch display screen by original user's long-press touch operation and is remained stationary as, therefore corresponding AR gesture operations can
To be to stretch out a finger, moved along Z-direction, then keep interval of time motionless, i.e., formed perpendicular to X-axis and Y-axis
Plane (i.e. display screen where plane) it is mobile, then keep interval of time motionless.
For above two AR gesture operations, its main feature is that finger is moved along Z-direction, it is clicking operation finger
Move always, and long-press operation is remained stationary as after finger moves a distance.Therefore terminal device can calculate adjacent gesture
Displacement distance of the palm key point along Z-direction between image.
Wherein, since palm key point includes multiple points, between adjacent images of gestures the movement of palm key point away from
From an array representation can be used.The crucial points of palm as shown in above-mentioned Figure 1B are 21, then hand between adjacent images of gestures
The displacement distance array of the palm key point along Z-direction has 21 elements.Carried out below by taking the palm key point shown in Figure 1B as an example
Explanation：
Images of gestures 1 is adjacent with images of gestures 2, and the time of images of gestures 1 is earlier than the time of images of gestures 2, gesture figure
As 1 palm key point in centre of the palm coordinate be (a1, b1, c1), in the palm key point of images of gestures 2 centre of the palm coordinate be (a2,
B2, c2), then the displacement distance of the centre of the palm along Z-direction is d1=c1-c2 in palm key point.And so on, images of gestures 1 with
The displacement distance of other in palm key point along Z-direction is d2~d21 between images of gestures 2.
Step 203：First variable is determined as to the gesture model data of the first species.
From above-mentioned steps 202, the gesture model data for the first species being made of the first variable are including at least one
Displacement distance array.For example, the images of gestures of collection is 10 frames, then gesture model data include 9 displacement distance arrays.And
The gesture model data of first species can include two kinds of AR gesture operations：Click and long-press.
It will be appreciated by persons skilled in the art that the gesture model data of the first species be not limited to only click and
Long-press touch operation, can also include other touch operations, for example, pushing button operation, the application will not enumerate herein.
In an exemplary scenario, for AR gesture operations be long-press gesture model data, by finger move one section away from
The characteristics of being remained stationary as from after understands that each element in last several displacement distance arrays in gesture model data should be equal
For 0.And be the gesture model data clicked on for AR gesture operations, the characteristics of one or many mobile, is carried out from finger,
Each element should have numerical value in each displacement distance array in gesture model data.
In the present embodiment, coordinate of the terminal device according to the palm key point per frame images of gestures in space coordinates is true
Displacement distance (first variable) of the palm key point along Z-direction between fixed adjacent images of gestures, and the first variable is determined as
The gesture model data of first species.The application is by will be used as the gesture model of the first species along the first variable of Z-direction
Data, can distinguish the click in AR gesture operations and long-press.
Fig. 3 is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment,
The present embodiment utilizes the above method provided by the embodiments of the present application, how to determine based on palm key point the gesture of preset kind
It is illustrative exemplified by model data, as shown in figure 3, including the following steps：
Step 301：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis.
The description of step 301 may refer to the description of above-mentioned steps 201, and details are not described herein.
Step 302：Determined according to coordinate of the palm key point of every frame images of gestures in the space coordinates adjacent
Second variable of palm key point between images of gestures, second variable between adjacent images of gestures palm key point in X
Axis and the displacement distance in the plane of Y-axis composition.
In one embodiment, sliding touch operation by original user is touched with one or more fingers (or the palm opened)
Display screen is touched, and a distance is moved to a direction, therefore, corresponding AR gesture operations can be with one or more fingers
(or the palm opened) moves in X-axis and the plane of Y-axis composition (i.e. plane where display screen) to a direction.In addition, by
It is with one or more fingers (or the palm opened) touch display screens that original user, which gently sweeps touch operation, and is moved to a direction
Dynamic relatively short distance, therefore corresponding AR gesture operations can be in X-axis and Y-axis with one or more fingers (or the palm opened)
In the plane of composition relatively short distance is moved to a direction.
For above two AR gesture operations, its main feature is that finger is moved in the plane that X-axis is formed with Y-axis, only
It is gently to sweep the mobile relatively short distance of operation.Therefore palm key point is in X-axis and Y between terminal device can calculate adjacent images of gestures
Displacement distance in the plane of axis composition.
Wherein, the associated description on displacement distance array may refer to the associated description of above-mentioned steps 202, herein no longer
Repeat.Illustrated below by taking the palm key point shown in Figure 1B as an example：
Images of gestures 1 is adjacent with images of gestures 2, and the time of images of gestures 1 is earlier than the time of images of gestures 2, gesture figure
As 1 palm key point in centre of the palm coordinate be (a1, b1, c1), in the palm key point of images of gestures 2 centre of the palm coordinate be (a2,
B2, c2), then between images of gestures 1 and images of gestures 2, shifting of the centre of the palm in X-axis and the plane of Y-axis composition in palm key point
Dynamic distance isAnd so on, between images of gestures 1 and images of gestures 2 in palm key point
Other X-axis and Y-axis composition plane on displacement distance be e2~e21.
Step 303：Second variable is determined as to the gesture model data of second species.
From above-mentioned steps 302, the gesture model data of second species can include two kinds of AR gesture operations：Slide and
Gently sweep.
It is the gesture model data gently swept for AR gesture operations, by finger movement compared with short distance in an exemplary scenario
From the characteristics of understand, each element in last several displacement distance arrays in gesture model data should be 0.And for
AR gesture operations for slide gesture model data, from finger to a direction move the characteristics of knowable to, in gesture model data
Each element should have numerical value in each displacement distance array.
In the present embodiment, coordinate of the terminal device according to the palm key point per frame images of gestures in space coordinates is true
Displacement distance (second variable) of the palm key point in X-axis and the plane of Y-axis composition between fixed adjacent images of gestures, and by the
Two variables are determined as the gesture model data of second species.The application passes through the second variable in the plane that forms X-axis and Y-axis
As the gesture model data of second species, the slip in AR gesture operations can be distinguished and gently swept.
Fig. 4 is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment,
The present embodiment utilizes the above method provided by the embodiments of the present application, how to determine based on palm key point the gesture of preset kind
It is illustrative exemplified by model data, as shown in figure 4, including the following steps：
Step 401：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis.
The description of step 401 may refer to the description of above-mentioned steps 201, and details are not described herein.
Step 402：Determined according to coordinate of the palm key point of every frame images of gestures in the space coordinates adjacent
The ternary of palm key point between images of gestures, ternary hand in palm key point between adjacent images of gestures
Refer to rotational angle of the key point with respect to centre of the palm key point.
In one embodiment, by original user go to touch operation be with two or more finger touch display screens, and with
Move clockwise or counterclockwise, therefore, corresponding AR gesture operations can be with two or more fingers with clockwise or
It is counterclockwise mobile.For above-mentioned AR gesture operations, its main feature is that finger to move clockwise or counterclockwise.Therefore terminal is set
It is standby to calculate between adjacent images of gestures rotational angle of the finger key point with respect to centre of the palm key point in palm key point.
Wherein, due in palm key point finger key point can include multiple points, hand between adjacent images of gestures
Finger key point can use an array representation with respect to the rotational angle of centre of the palm key point in palm key point.As shown in above-mentioned Figure 1B
The crucial points of finger have 14, then between adjacent images of gestures in palm key point finger key point with respect to centre of the palm key point
Rotational angle array has 14 elements.
Step 403：The ternary is determined as to the gesture model data of the third class.
From above-mentioned steps 402, the gesture model data for the third class being made of ternary are including at least one
Rotational angle array.For example, the images of gestures of collection is 10 frames, then gesture model data include 9 rotational angle arrays.And
The AR gesture operations of the gesture model data of the third class can include rotating.
In the present embodiment, coordinate of the terminal device according to the palm key point per frame images of gestures in space coordinates is true
Between fixed adjacent images of gestures in palm key point finger key point with respect to centre of the palm key point rotational angle (ternary), and
Ternary is determined as to the gesture model data of the third class.Ternary by being used as the gesture of the third class by the application
Model data, can identify the rotation in AR gesture operations.
Fig. 5 is the embodiment flow chart of another man-machine interaction method of the application according to an exemplary embodiment,
The present embodiment utilizes the above method provided by the embodiments of the present application, how to determine based on palm key point the gesture of preset kind
It is illustrative exemplified by model data, as shown in figure 5, including the following steps：
Step 501：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis.
The description of step 501 may refer to the description of above-mentioned steps 201, and details are not described herein.
Step 502：Determined according to coordinate of the palm key point of every frame images of gestures in the space coordinates adjacent
4th variable of palm key point between images of gestures, the 4th variable hand in palm key point between adjacent images of gestures
Refer to the distance change amount of key point and centre of the palm key point.
In one embodiment, with two or more finger touch display screens, then it is to shrink touch operation by original user
Finger is close together, therefore, corresponding AR gesture operations can be moved to two or more fingers by stretched condition
Closed configuration.In addition, by original user stretch touch operation be use two or more finger touch display screens, then by finger divide
Open, therefore corresponding AR gesture operations can move to stretched condition by closed configuration with two or more fingers.
For above two AR gesture operations, its feature all refers to finger stretched condition and closed configuration, therefore, finger
The distance between centre of the palm is different under different conditions, and palm is crucial between terminal device can calculate adjacent images of gestures
The distance change amount of finger key point and centre of the palm key point in point.
In an exemplary scenario, images of gestures 1 is adjacent with images of gestures 2, and the time of images of gestures 1 is earlier than images of gestures
2 time, the coordinate in the centre of the palm is (a1, b1, c1) in the palm key point of images of gestures 1, and the coordinate of a key point of middle finger is
(x1, y1, z1), the coordinate in the centre of the palm is (a2, b2, c2) in the palm key point of images of gestures 2, a key of corresponding middle finger
The coordinate of point is (x2, y2, z2), then between images of gestures 1 and images of gestures 2, finger key point is closed with the centre of the palm in palm key point
The distance change amount of key point is：
Step 503：4th variable is determined as to the gesture model data of the 4th species.
Described in above-mentioned steps 502, the gesture model data of the 4th species can include two kinds of AR gesture operations：Receive
Contracting and stretching.
It will be appreciated by persons skilled in the art that the gesture model data of the 4th species be not limited to only stretching and
Touch operation is shunk, other touch operations can also be included, for example, waiting (finger is always maintained at closed configuration) to be operated, this Shen
Please it will not enumerate herein.
It is the gesture model data shunk for AR gesture operations, finger is by stretched condition in an exemplary scenario
To closed configuration change the characteristics of understand, the distance change amount array numerical value in gesture model data should be gradually reduced.And
The characteristics of gesture model data for AR gesture operations for stretching, finger is the change by closed configuration to stretched condition, can
Know, should gradually increase apart from variable quantity array numerical value in gesture model data.
In the present embodiment, coordinate of the terminal device according to the palm key point per frame images of gestures in space coordinates is true
The distance change amount (the 4th variable) of finger key point and centre of the palm key point in palm key point between fixed adjacent images of gestures, and
4th variable is determined as to the gesture model data of the 4th species.4th variable by being used as the gesture of the 4th species by the application
Model data, can distinguish stretching and contraction in AR gesture operations.
It will be appreciated by persons skilled in the art that the first species gesture model number in above-mentioned Fig. 2A illustrated embodiments
According to the second species gesture model data in, embodiment illustrated in fig. 3, the third class gesture model number in embodiment illustrated in fig. 4
The 4th species model data according to this and in embodiment illustrated in fig. 5, can travel through to obtain in the embodiment shown in Figure 1A,
One or more therein can be traveled through, based on different use demands, different types of gesture model data can be traveled through, often
The gesture model data of one species can be used for distinguishing one or more AR gesture operations.
Corresponding with the embodiment of foregoing man-machine interaction method, present invention also provides the embodiment of human-computer interaction device.
The embodiment of the application human-computer interaction device can be applied on the terminal device.Device embodiment can pass through software
Realize, can also be realized by way of hardware or software and hardware combining.Exemplified by implemented in software, as on a logical meaning
Device, be in by the processor of equipment where it, corresponding computer program instructions in nonvolatile memory are read
Deposit what middle operation was formed.For hardware view, as shown in fig. 6, exemplified for the application according to embodiment implementation one
The hardware structure diagram of kind of terminal device, except the processor shown in Fig. 6, memory, network interface and nonvolatile memory it
Outside, the equipment in embodiment where device can also include other hardware, to this no longer generally according to the actual functional capability of the equipment
Repeat.
Fig. 7 is a kind of example structure figure of human-computer interaction device of the application according to an exemplary embodiment, such as
Shown in Fig. 7, which includes：Gather extraction module 610, determining module 620, matching module 630 and perform friendship
Mutual module 640.
Extraction module 610 is gathered, for gathering one group of continuous images of gestures in the range of pre-set space, and described in extraction
The palm key point of images of gestures；
Determining module 620, for determining the gesture model data of preset kind based on the palm key point；
Matching module 630, for gesture model data template data corresponding with the preset kind to be matched；
Interactive module 640 is performed, the corresponding touch action of the template data for being obtained according to matching performs interactive operation.
In an optional implementation, the determining module 620, specifically for establishing space coordinates, the space
Coordinate system includes X-axis, Y-axis and Z axis；It is true according to coordinate of the palm key point of every frame images of gestures in the space coordinates
First variable of palm key point between fixed adjacent images of gestures, first variable palm between adjacent images of gestures are crucial
Displacement distance of the point along Z-direction；First variable is determined as to the gesture model data of the first species.
In an optional implementation, the determining module 620, specifically for establishing space coordinates, the space
Coordinate system includes X-axis, Y-axis and Z axis；It is true according to coordinate of the palm key point of every frame images of gestures in the space coordinates
Second variable of palm key point between fixed adjacent images of gestures, second variable palm between adjacent images of gestures are crucial
Displacement distance of the point in X-axis and the plane of Y-axis composition；Second variable is determined as to the gesture model number of second species
According to.
In an optional implementation, the determining module 620, specifically for establishing space coordinates, the space
Coordinate system includes X-axis, Y-axis and Z axis；It is true according to coordinate of the palm key point of every frame images of gestures in the space coordinates
The ternary of palm key point between fixed adjacent images of gestures, ternary palm between adjacent images of gestures are crucial
Rotational angle of the finger key point with respect to centre of the palm key point in point；The ternary is determined as to the gesture model of the third class
Data.
In an optional implementation, the determining module 620, specifically for establishing space coordinates, the space
Coordinate system includes X-axis, Y-axis and Z axis；It is true according to coordinate of the palm key point of every frame images of gestures in the space coordinates
4th variable of palm key point between fixed adjacent images of gestures, the 4th variable palm between adjacent images of gestures are crucial
The distance change amount of finger key point and centre of the palm key point in point；4th variable is determined as to the gesture model of the 4th species
Data.
The function of unit and effect realizes that process specifically refers to step is corresponded in the above method in above device
Realize process, details are not described herein.
For device embodiment, since it corresponds essentially to embodiment of the method, so related part is real referring to method
Apply the part explanation of example.Device embodiment described above is only schematical, wherein described be used as separating component
The unit of explanation may or may not be physically separate, can be as the component that unit is shown or can also
It is not physical location, you can with positioned at a place, or can also be distributed in multiple network unit.Can be according to reality
Need to select some or all of module therein to realize the purpose of application scheme.Those of ordinary skill in the art are not paying
In the case of going out creative work, you can to understand and implement.
The foregoing is merely the preferred embodiment of the application, not limiting the application, all essences in the application
God and any modification, equivalent substitution, improvement and etc. within principle, done, should be included within the scope of the application protection.
Claims (10)
- A kind of 1. man-machine interaction method, it is characterised in that the described method includes：One group of continuous images of gestures is gathered in the range of pre-set space, and extracts the palm key point of the images of gestures；The gesture model data of preset kind are determined based on the palm key point；Gesture model data template data corresponding with the preset kind is matched；The corresponding touch action of template data obtained according to matching performs interactive operation.
- 2. according to the method described in claim 1, it is characterized in that, described determine preset kind based on the palm key point Gesture model data, including：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis；Determined according to coordinate of the palm key point of every frame images of gestures in the space coordinates between adjacent images of gestures First variable of palm key point, first variable palm key point moving along Z-direction between adjacent images of gestures Distance；First variable is determined as to the gesture model data of the first species.
- 3. according to the method described in claim 1, it is characterized in that, described determine preset kind based on the palm key point Gesture model data, including：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis；Determined according to coordinate of the palm key point of every frame images of gestures in the space coordinates between adjacent images of gestures Second variable of palm key point, second variable palm key point between adjacent images of gestures are formed in X-axis and Y-axis Displacement distance in plane；Second variable is determined as to the gesture model data of second species.
- 4. according to the method described in claim 1, it is characterized in that, described determine preset kind based on the palm key point Gesture model data, including：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis；Determined according to coordinate of the palm key point of every frame images of gestures in the space coordinates between adjacent images of gestures The ternary of palm key point, finger key point is opposite in palm key point between adjacent images of gestures for the ternary The rotational angle of centre of the palm key point；The ternary is determined as to the gesture model data of the third class.
- 5. according to the method described in claim 1, it is characterized in that, described determine preset kind based on the palm key point Gesture model data, including：Space coordinates are established, the space coordinates include X-axis, Y-axis and Z axis；Determined according to coordinate of the palm key point of every frame images of gestures in the space coordinates between adjacent images of gestures 4th variable of palm key point, the 4th variable between adjacent images of gestures in palm key point finger key point with the palm The distance change amount of heart key point；4th variable is determined as to the gesture model data of the 4th species.
- 6. a kind of human-computer interaction device, it is characterised in that described device includes：Extraction module is gathered, for gathering one group of continuous images of gestures in the range of pre-set space, and extracts the gesture figure The palm key point of picture；Determining module, for determining the gesture model data of preset kind based on the palm key point；Matching module, for gesture model data template data corresponding with the preset kind to be matched；Interactive module is performed, the corresponding touch action of the template data for being obtained according to matching performs interactive operation.
- 7. device according to claim 6, it is characterised in that the determining module, specifically for establishing space coordinates, The space coordinates include X-axis, Y-axis and Z axis；According to the palm key point of every frame images of gestures in the space coordinates Coordinate determine the first variable of palm key point between adjacent images of gestures, first variable is between adjacent images of gestures Displacement distance of the palm key point along Z-direction；First variable is determined as to the gesture model data of the first species.
- 8. device according to claim 6, it is characterised in that the determining module, specifically for establishing space coordinates, The space coordinates include X-axis, Y-axis and Z axis；According to the palm key point of every frame images of gestures in the space coordinates Coordinate determine the second variable of palm key point between adjacent images of gestures, second variable is between adjacent images of gestures Displacement distance of the palm key point in X-axis and the plane of Y-axis composition；Second variable is determined as to the gesture of second species Model data.
- 9. device according to claim 6, it is characterised in that the determining module, specifically for establishing space coordinates, The space coordinates include X-axis, Y-axis and Z axis；According to the palm key point of every frame images of gestures in the space coordinates Coordinate determine the ternary of palm key point between adjacent images of gestures, the ternary is between adjacent images of gestures Rotational angle of the finger key point with respect to centre of the palm key point in palm key point；The ternary is determined as the third class Gesture model data.
- 10. device according to claim 6, it is characterised in that the determining module, specifically for establishing space coordinate System, the space coordinates include X-axis, Y-axis and Z axis；According to the palm key point of every frame images of gestures in the space coordinate Coordinate in system determines the 4th variable of palm key point between adjacent images of gestures, and the 4th variable is adjacent images of gestures Between in palm key point finger key point and centre of the palm key point distance change amount；4th variable is determined as the 4th kind The gesture model data of class.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711391520.XA CN107967061A (en) | 2017-12-21 | 2017-12-21 | Man-machine interaction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711391520.XA CN107967061A (en) | 2017-12-21 | 2017-12-21 | Man-machine interaction method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107967061A true CN107967061A (en) | 2018-04-27 |
Family
ID=61995723
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711391520.XA Pending CN107967061A (en) | 2017-12-21 | 2017-12-21 | Man-machine interaction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107967061A (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108829253A (en) * | 2018-06-19 | 2018-11-16 | 北京科技大学 | A kind of analog music commander's playback method and device |
| CN109710081A (en) * | 2019-02-20 | 2019-05-03 | 杭州老板电器股份有限公司 | Gesture induction test device and method |
| CN110019688A (en) * | 2019-01-23 | 2019-07-16 | 艾肯特公司 | The method that robot is trained |
| CN110321008A (en) * | 2019-06-28 | 2019-10-11 | 北京百度网讯科技有限公司 | Exchange method, device, equipment and storage medium based on AR model |
| CN110962797A (en) * | 2019-12-27 | 2020-04-07 | 上海昶枫科技有限公司 | Automobile electronic steering column lock control system and method |
| CN110979214A (en) * | 2019-12-27 | 2020-04-10 | 上海昶枫科技有限公司 | Automobile electronic control system |
| CN111025999A (en) * | 2019-12-27 | 2020-04-17 | 上海昶枫科技有限公司 | Energy-saving control process for automobile electronic controller |
| CN111045521A (en) * | 2019-12-27 | 2020-04-21 | 上海昶枫科技有限公司 | Automobile electronic device control system and control method |
| WO2020088069A1 (en) * | 2018-10-30 | 2020-05-07 | 北京达佳互联信息技术有限公司 | Hand gesture keypoints detection method and apparatus, electronic device, and storage medium |
| CN111143217A (en) * | 2019-12-27 | 2020-05-12 | 上海昶枫科技有限公司 | Automobile electronic control unit simulation system |
| CN111221406A (en) * | 2018-11-23 | 2020-06-02 | 杭州萤石软件有限公司 | Information interaction method and device |
| CN111885406A (en) * | 2020-07-30 | 2020-11-03 | 深圳创维－Rgb电子有限公司 | Smart television control method and device, rotatable television and readable storage medium |
| CN111898489A (en) * | 2020-07-15 | 2020-11-06 | 北京百度网讯科技有限公司 | Method and device for marking palm pose, electronic equipment and storage medium |
| CN112121280A (en) * | 2020-08-31 | 2020-12-25 | 浙江大学 | Control method and control system of heart sound box |
| CN112383805A (en) * | 2020-11-16 | 2021-02-19 | 四川长虹电器股份有限公司 | Method for realizing man-machine interaction at television end based on human hand key points |
| CN112613384A (en) * | 2020-12-18 | 2021-04-06 | 安徽鸿程光电有限公司 | Gesture recognition method, gesture recognition device and control method of interactive display equipment |
| CN113384291A (en) * | 2021-06-11 | 2021-09-14 | 北京华医共享医疗科技有限公司 | Medical ultrasonic detection method and system |
| WO2022179376A1 (en) * | 2021-02-24 | 2022-09-01 | Oppo广东移动通信有限公司 | Gesture control method and apparatus, and electronic device and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103472916A (en) * | 2013-09-06 | 2013-12-25 | 东华大学 | Man-machine interaction method based on human body gesture recognition |
| WO2014001058A1 (en) * | 2012-06-25 | 2014-01-03 | Softkinetic Software | Improvements in or relating to three dimensional close interactions |
| CN104123007A (en) * | 2014-07-29 | 2014-10-29 | 电子科技大学 | Multidimensional weighted 3D recognition method for dynamic gestures |
| CN104331158A (en) * | 2014-10-29 | 2015-02-04 | 山东大学 | Gesture-controlled human-computer interaction method and device |
| CN104808788A (en) * | 2015-03-18 | 2015-07-29 | 北京工业大学 | Method for controlling user interfaces through non-contact gestures |
| CN104866824A (en) * | 2015-05-17 | 2015-08-26 | 华南理工大学 | Manual alphabet identification method based on Leap Motion |
| CN105589553A (en) * | 2014-09-23 | 2016-05-18 | 上海影创信息科技有限公司 | Gesture control method and system for intelligent equipment |
| CN107092349A (en) * | 2017-03-20 | 2017-08-25 | 重庆邮电大学 | A kind of sign Language Recognition and method based on RealSense |
| CN107403167A (en) * | 2017-08-03 | 2017-11-28 | 华中师范大学 | Gesture identification method and device |
-
2017
- 2017-12-21 CN CN201711391520.XA patent/CN107967061A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014001058A1 (en) * | 2012-06-25 | 2014-01-03 | Softkinetic Software | Improvements in or relating to three dimensional close interactions |
| CN103472916A (en) * | 2013-09-06 | 2013-12-25 | 东华大学 | Man-machine interaction method based on human body gesture recognition |
| CN104123007A (en) * | 2014-07-29 | 2014-10-29 | 电子科技大学 | Multidimensional weighted 3D recognition method for dynamic gestures |
| CN105589553A (en) * | 2014-09-23 | 2016-05-18 | 上海影创信息科技有限公司 | Gesture control method and system for intelligent equipment |
| CN104331158A (en) * | 2014-10-29 | 2015-02-04 | 山东大学 | Gesture-controlled human-computer interaction method and device |
| CN104808788A (en) * | 2015-03-18 | 2015-07-29 | 北京工业大学 | Method for controlling user interfaces through non-contact gestures |
| CN104866824A (en) * | 2015-05-17 | 2015-08-26 | 华南理工大学 | Manual alphabet identification method based on Leap Motion |
| CN107092349A (en) * | 2017-03-20 | 2017-08-25 | 重庆邮电大学 | A kind of sign Language Recognition and method based on RealSense |
| CN107403167A (en) * | 2017-08-03 | 2017-11-28 | 华中师范大学 | Gesture identification method and device |
Non-Patent Citations (1)
| Title |
|---|
| 田启川: "《虹膜识别原理及算法》", 30 June 2010 * |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108829253B (en) * | 2018-06-19 | 2021-06-01 | 北京科技大学 | Simulated music command playing method and device |
| CN108829253A (en) * | 2018-06-19 | 2018-11-16 | 北京科技大学 | A kind of analog music commander's playback method and device |
| US11514706B2 (en) | 2018-10-30 | 2022-11-29 | Beijing Dajia Internet Information Technology Co., Ltd. | Method and device for detecting hand gesture key points |
| WO2020088069A1 (en) * | 2018-10-30 | 2020-05-07 | 北京达佳互联信息技术有限公司 | Hand gesture keypoints detection method and apparatus, electronic device, and storage medium |
| CN111221406A (en) * | 2018-11-23 | 2020-06-02 | 杭州萤石软件有限公司 | Information interaction method and device |
| CN111221406B (en) * | 2018-11-23 | 2023-10-13 | 杭州萤石软件有限公司 | Information interaction method and device |
| CN110019688A (en) * | 2019-01-23 | 2019-07-16 | 艾肯特公司 | The method that robot is trained |
| CN109710081A (en) * | 2019-02-20 | 2019-05-03 | 杭州老板电器股份有限公司 | Gesture induction test device and method |
| CN110321008A (en) * | 2019-06-28 | 2019-10-11 | 北京百度网讯科技有限公司 | Exchange method, device, equipment and storage medium based on AR model |
| CN110321008B (en) * | 2019-06-28 | 2023-10-24 | 北京百度网讯科技有限公司 | Interaction method, device, equipment and storage medium based on AR model |
| CN111025999A (en) * | 2019-12-27 | 2020-04-17 | 上海昶枫科技有限公司 | Energy-saving control process for automobile electronic controller |
| CN111143217A (en) * | 2019-12-27 | 2020-05-12 | 上海昶枫科技有限公司 | Automobile electronic control unit simulation system |
| CN111045521A (en) * | 2019-12-27 | 2020-04-21 | 上海昶枫科技有限公司 | Automobile electronic device control system and control method |
| CN110979214A (en) * | 2019-12-27 | 2020-04-10 | 上海昶枫科技有限公司 | Automobile electronic control system |
| CN110962797A (en) * | 2019-12-27 | 2020-04-07 | 上海昶枫科技有限公司 | Automobile electronic steering column lock control system and method |
| CN111898489A (en) * | 2020-07-15 | 2020-11-06 | 北京百度网讯科技有限公司 | Method and device for marking palm pose, electronic equipment and storage medium |
| CN111898489B (en) * | 2020-07-15 | 2023-08-08 | 北京百度网讯科技有限公司 | Method and device for marking palm pose, electronic equipment and storage medium |
| CN111885406A (en) * | 2020-07-30 | 2020-11-03 | 深圳创维－Rgb电子有限公司 | Smart television control method and device, rotatable television and readable storage medium |
| CN112121280A (en) * | 2020-08-31 | 2020-12-25 | 浙江大学 | Control method and control system of heart sound box |
| CN112383805A (en) * | 2020-11-16 | 2021-02-19 | 四川长虹电器股份有限公司 | Method for realizing man-machine interaction at television end based on human hand key points |
| CN112613384A (en) * | 2020-12-18 | 2021-04-06 | 安徽鸿程光电有限公司 | Gesture recognition method, gesture recognition device and control method of interactive display equipment |
| CN112613384B (en) * | 2020-12-18 | 2023-09-19 | 安徽鸿程光电有限公司 | Gesture recognition method, gesture recognition device and control method of interactive display equipment |
| WO2022179376A1 (en) * | 2021-02-24 | 2022-09-01 | Oppo广东移动通信有限公司 | Gesture control method and apparatus, and electronic device and storage medium |
| CN113384291A (en) * | 2021-06-11 | 2021-09-14 | 北京华医共享医疗科技有限公司 | Medical ultrasonic detection method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107967061A (en) | Man-machine interaction method and device | |
| EP2511812B1 (en) | Continuous recognition method of multi-touch gestures from at least two multi-touch input devices | |
| Nai et al. | Fast hand posture classification using depth features extracted from random line segments | |
| Murthy et al. | Hand gesture recognition using neural networks | |
| Park | A method for controlling the mouse movement using a real time camera | |
| MX2009000305A (en) | Virtual controller for visual displays. | |
| KR20170009979A (en) | Methods and systems for touch input | |
| EP1730627A2 (en) | System and method for inputing user commands to a processor | |
| CN106774850A (en) | A kind of mobile terminal and its interaction control method | |
| CN106843669A (en) | Application interface operating method and device | |
| Shah et al. | Hand gesture based user interface for computer using a camera and projector | |
| Wilson et al. | Flowmouse: A computer vision-based pointing and gesture input device | |
| Molina et al. | A natural and synthetic corpus for benchmarking of hand gesture recognition systems | |
| CN112199015B (en) | Intelligent interaction all-in-one machine and writing method and device thereof | |
| Tosas et al. | Virtual touch screen for mixed reality | |
| CN112578987A (en) | Off-screen interactive touch all-in-one machine and interaction method thereof | |
| CN106648423A (en) | Mobile terminal and interactive control method thereof | |
| CN110414393A (en) | A kind of natural interactive method and terminal based on deep learning | |
| Dhamanskar et al. | Human computer interaction using hand gestures and voice | |
| Triki et al. | Imaging and HMI: Fondations and complementarities | |
| Chandra Gope | Hand tracking and hand gesture recognition for human computer interaction | |
| CN114967927B (en) | Intelligent gesture interaction method based on image processing | |
| Jayasathyan et al. | Implementation of Real Time Virtual Clicking using OpenCV | |
| Lee et al. | Finger controller: Natural user interaction using finger gestures | |
| El Magrouni et al. | Approach for the construction of gestural interfaces to control graphical interfaces based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180427 |
|
| RJ01 | Rejection of invention patent application after publication |