CN107272893B - Man-machine interaction system and method for controlling non-touch screen based on gestures - Google Patents
Man-machine interaction system and method for controlling non-touch screen based on gestures Download PDFInfo
- Publication number
- CN107272893B CN107272893B CN201710414437.3A CN201710414437A CN107272893B CN 107272893 B CN107272893 B CN 107272893B CN 201710414437 A CN201710414437 A CN 201710414437A CN 107272893 B CN107272893 B CN 107272893B
- Authority
- CN
- China
- Prior art keywords
- hand
- shape
- palm
- display screen
- contour
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/145—Illumination specially adapted for pattern recognition, e.g. using gratings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Artificial Intelligence (AREA)
- Human Computer Interaction (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Image Analysis (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention relates to a human-computer interaction system and method based on a gesture control non-touch screen. The system arranges a polaroid in front of a high-speed camera, and is used for eliminating polarized light on a computer or television display screen by adjusting the rotation of the polaroid to a proper angle, so that the noise influence in a video is removed, and the hand part in the video is highlighted. The method extracts the outer contour of the hand part and recognizes the gesture by using the shape space theory method, so that the computer and the intelligent television can be operated according to the recognized gesture change, and man-machine interaction is realized. The invention can reduce the hardware cost of the traditional human-computer interaction, can be used in various human-computer interaction fields such as intelligent cities, intelligent families, large-scale exhibition facility control, intelligent lecture report display, electronic entertainment and the like, and can improve the life quality of people, particularly the old and the disabled.
Description
Technical Field
The invention provides a human-computer interaction system and method based on a gesture control non-touch screen. The invention can reduce the hardware cost of the traditional human-computer interaction, can be applied to various human-computer interaction fields such as intelligent cities, intelligent families, large-scale exhibition facility control, intelligent lecture report display, electronic entertainment and the like, and can improve the life quality of people, particularly the old and the disabled.
Technical Field
With the rapid development of society, people tend to operate and control machines using comfortable, convenient, and safe methods. The intelligent Human-Computer Interaction (HCI) technology has been widely applied in the fields of intelligent home, entertainment facilities, terminal electronic equipment and the like; in smart phones, motion sensing game facilities and propaganda and display equipment, an intelligent interaction technology has an extremely important and even decisive position; in the fields of medical rehabilitation and intelligent household appliances, the intelligent interaction technology is rapidly popularized and applied. The man-machine interaction technology is an important entry point for building smart cities and smart families, enriches intelligent elements in modern society, meets the living and entertainment requirements of people and is an important entry point.
The invention utilizes gesture change to operate the intelligent machine, and realizes simple and rapid human-computer interaction. Different commands are given to the computer and the intelligent television by recognizing different gestures, and a man-machine interaction system for controlling the computer or the television based on gesture changes is established. The invention can reduce the hardware cost of the traditional human-computer interaction, strives to provide fresh and convenient experience for users, and can be used in various human-computer interaction fields such as intelligent cities, intelligent families, large-scale exhibition facility control, intelligent presentation of lecture reports, electronic entertainment and the like. The intelligent machine is not only suitable for general users, but also suitable for disabled people with inconvenient body or old people who do not have the energy to learn and are familiar with traditional instruction usage, and the disabled people and the old people can operate the intelligent machine through gestures in a simple and convenient way, so that the life quality of the disabled people is improved, the labor cost of caregivers is reduced, and the burden of family members is reduced.
Disclosure of Invention
The invention aims to provide a human-computer interaction system and method based on a gesture control non-touch screen, aiming at the defects of the prior art. The gesture is obtained by using the high-speed camera, polarized light on a display screen of a computer or a television is eliminated by using a polarizing film arranged in front of the camera, and the gesture is correctly recognized, so that the computer and the intelligent television are operated.
In order to achieve the purpose, the invention adopts the following technical scheme:
a human-computer interaction system based on a gesture control non-touch screen comprises a main controller and a display screen; the display screen is vertically arranged; placing a high-speed camera obliquely above the display screen and facing the display screen; a polarizing plate is arranged in front of the camera; eliminating polarized light on the display screen by adjusting the rotation angle of the polaroid; the display and the high-speed camera are connected to the main controller; the display screen can rotate in three-dimensional space, and the shooting position and angle of the camera can be changed correspondingly.
A human-computer interaction system based on a gesture control non-touch screen comprises a main controller and a display screen; the display screen is horizontally placed on a desktop; a high-speed camera perpendicular to the display screen is arranged right above the display screen; a polarizing plate is arranged in front of the camera; eliminating polarized light on the display screen by adjusting the rotation angle of the polaroid; a display screen and a high speed camera are connected to the main controller.
The main controller analyzes and calculates the pictures acquired by the high-speed camera, runs corresponding algorithms and outputs the results to the display screen. The display screen is used for displaying the result of the man-machine interaction of the operation of the main controller, and can be vertically placed or horizontally placed according to different requirements, and can also be obliquely placed when necessary. The high speed camera is used to capture images of the gesture and transmit the images to the master controller. The polaroid is placed in front of the camera, and the polaroid can eliminate polarized light emitted on the display screen by adjusting the rotating angle of the polaroid, so that the shape of a human hand in a video can be extracted.
A human-computer interaction method based on a gesture control non-touch screen is operated by adopting the system, and is characterized by comprising the following operation steps:
1. starting a human-computer interaction system:
the high-speed camera switch and the display screen switch are turned on, then the main controller is started to control the operation of the high-speed camera and the display screen, and light emitted by the display screen is filtered out through the polaroid.
2. Setting a working range in the human-computer interaction system:
the invention stipulates that the display screen is taken as a background object, only the range of the space position of the display screen is taken as a man-machine interaction working range in a shot image, and objects outside the range are ignored; since the positions of the image pickup apparatus and the display screen are relatively fixed, it can be used for a long time by one setting in the program.
3. The gesture recognition is realized in the man-machine interaction system:
the man-machine interaction in the invention is based on gesture recognition, generally speaking, any gesture recognition algorithm, such as Hu method, SVM method, deep learning method to recognize gestures, can be applied to the invention; the gesture recognition adopted by the invention is a method based on a shape space theory, and is specifically described as follows:
(3-1) obtaining an image of a foreground human body:
a user places a handle in a working range area of a human-computer interaction system, and obtains a video image containing the handle through a high-speed camera; removing light emitted by a display screen by adjusting the rotation angle of a polaroid arranged in front of a camera to obtain an image of a foreground human body;
(3-2) obtaining the shape of the hand:
the color of the image is converted from the traditional RGB format to the HSV format, so that the difference between the skin color and other colors can be increased, the human body information containing the skin is extracted, the information contains the image information of the hand, partial noise is removed by using a Gaussian denoising method and a mathematical morphology method, and correct information in the hand image is reserved;
(3-3) converting the video image containing the hand information into a binary image;
(3-4) extracting the hand outline in the image:
the object in the image contains an inner contour and an outer contour, where only the outer contour is extracted and the inner contour is eliminated; calculating the area of each outline, and if the area is too small, deleting the outline as noise, so that the hand outline is reserved in the whole image;
(3-5) extracting the palm center point:
the hand contour of the user may contain partial arm contour information, which may affect the recognition of gestures; the shortest distance from the palm center point to the outline is the longest in the shortest distances from the points on the center lines of all the shapes to the outline, so the palm center point can be found by utilizing the property; the formula for obtaining the palm center point is as follows:
p hand shape interior arbitrary points, psIs the point on the outer contour of the hand, | | p, psI means p and psEuclidean distance between; the central point p of the palm can be obtained by the formulac；
(3-6) extracting the external contour of the palm:
the shortest distance formula from the palm center point to the hand outer contour is as follows:
ds＝||pc-ps||， (2)
pcis the palm center point calculated by the formula (1)，psIs a point on the outer contour of the hand, dsIs the shortest distance from the center point of the palm to the outer contour of the hand; the hand shape is divided by drawing a circle by taking the length of twice the shortest distance as a radius and the center point of the palm as the center of the circle, so that the shape of the arm part can be cut off; sometimes, part of the shape of the finger is also cut away; because the shape of the arm extends to the outside of the working area in the working area, the shape which does not extend to the outside of the working area outside the circle is used as the finger shape, so that the finger shape is recovered to obtain the complete palm shape; if all connected shapes outside the circle do not extend outside the work area, the circle outer shape area is used to determine if it is part of the palm, and the formula is as follows:
wherein S isiIs the circle outline area, S, of the shape of the connecting circle inner handcIs the area of the circle, ζ is the threshold value, setting
The shape of i satisfying the formula (3) can be confirmed as a part of the palm;
(3-7) extracting palm contour characteristic points:
taking one of two cutting points of a palm and an arm as an initial characteristic point, taking the other one as an end characteristic point, starting from the initial characteristic point, taking N points along a contour line to the end characteristic point with equal arc length, taking M points from the end characteristic point to the initial characteristic point with equal arc length in the opposite direction, wherein the characteristic points on the contour line are N + M + 2; the more the feature points are, the more accurate the gesture recognition is; conversely, the fewer the feature points, the faster the speed; the specific number is determined by the user;
and (3-8) acquiring palm contour characteristic points in the database:
processing the hand image in the database in the steps (3-2) to (3-7) to obtain the characteristic points of the palm outline of the database; if the hand image in the database is the palm outline image after the steps from (3-2) to (3-6), the system only needs to adopt the step (3-7) according to the requirement of extracting the number of the feature points on the contour line during each operation;
(3-9) recognizing the gesture by using a shape space theory method:
a plurality of new shapes formed by scaling of one shape at different positions, rotation directions and different scales can be represented by one point in a shape space, and before gesture recognition is carried out by utilizing a shape space theory, palm contour feature points need to be projected into the shape space; two different shape similarities can be represented by Procrustean distance in shape space, which is expressed as follows in real number space:
dp[O(τ1),O(τ2)]＝inf[cos-1(＜x,y＞):x∈O(τ1),y∈O(τ2)]， (4)
wherein, tau1And τ2Respectively setting two palm outline feature point sets which need to be compared with similarity; tau is1And τ2Projecting the two vectors into a hypersphere, and marking the hypersphere as Pre-shape space; o (tau)1) And O (τ)2) Are each tau1And τ2A great circle in the Pre-shape space; < x, y > is the inner product between x and y;
in the case of complex space, Procrumean distance formula is as follows:
wherein, tau1jAnd τ2jAre each tau1And τ2The jth complex coordinate of (a); tau is*Is the complex conjugate of τ;
by formulas (4) and (5), the similarity of each standard hand and the observing hand in the database is calculated if their distance satisfies the following formula:
dp[O(τ1),O(τ2)]≤ξ, (6)
xi is a threshold value for comparing similarity of a standard hand and an observation hand, and may have different values according to different numbers of contour points, and xi is set to be 0.4;
there may be many standard hand contours satisfying the formula (6), and among these standard hand contours, the standard hand having the shortest Procrustean distance to the observer hand is used as the identification hand type of the observer hand;
if all the standard hands are traversed, the hand type of the observation hand is not recognized, and then the user is required to adjust the hand posture to inform that the recognition is failed; in addition, by this step, the shape of the face in the video can be eliminated.
4. Different gestures are selected according to different software functions to control the software:
the software is controlled and operated by different gestures according to different functions of the software, for example, the movement of the software window is controlled by two fingers, the movement of the thumb and the index finger of two hands is used for expanding, and the software window is enlarged or reduced.
5. The system is shut down.
Compared with the prior art, the invention has the following obvious and prominent substantive characteristics and remarkable technical progress:
the invention arranges a polaroid in front of the high-speed camera, and eliminates polarized light on the display screen through the polaroid, thereby helping to identify gestures, further operating a computer and an intelligent television according to gesture changes, and realizing man-machine interaction. The invention has low cost, is suitable for various human-computer interaction fields such as intelligent cities, intelligent families, large-scale exhibition facility control, intelligent lecture report display, electronic entertainment and the like, and can improve the life quality of people, particularly the old and the disabled.
Description of the drawings:
FIG. 1 is a schematic structural diagram of a human-computer interaction system of one type in the present invention:
the human-computer interaction system based on the gesture control non-touch screen comprises a main controller (1) and a display screen (2), and is characterized in that: the display screen (2) is vertically arranged; a high-speed camera (3) is arranged above the display screen (2) in an inclined way and faces the display screen (2); a polarizing plate (4) is arranged in front of the camera (3); eliminating polarized light on the display screen (2) by adjusting the rotation angle of the polaroid (4); the display (2) and the high-speed camera (3) are connected to the main controller (1); the display screen (2) can rotate in a three-dimensional space, and the shooting position and angle of the camera (3) can be changed correspondingly.
FIG. 2 is a schematic structural diagram of a human-computer interaction system of one type in the present invention:
the human-computer interaction system based on the gesture control non-touch screen comprises a main controller (1) and a display screen (2), and is characterized in that: the display screen (2) is horizontally placed on a desktop; a high-speed camera (3) vertical to the display screen (2) is arranged right above the display screen (2); a polarizing plate (4) is arranged in front of the camera (3); eliminating polarized light on the display screen (2) by adjusting the rotation angle of the polaroid (4); a display screen (2) and a high-speed camera (3) are connected to a main controller (1).
FIG. 3 is a workflow of human-computer interaction of the present invention.
FIG. 4 is a workflow of gesture recognition in the present invention.
The specific implementation mode is as follows:
the embodiments of the invention are described in detail below with reference to the accompanying drawings:
the first embodiment is as follows:
referring to fig. 1, the human-computer interaction system based on gesture control non-touch screen of the present invention comprises a main controller (1) and a display screen (2), and is characterized in that: the display screen (2) is vertically arranged; a high-speed camera (3) is arranged above the display screen (2) in an inclined way and faces the display screen (2); a polarizing plate (4) is arranged in front of the camera (3); eliminating polarized light on the display screen (2) by adjusting the rotation angle of the polaroid (4); the display (2) and the high-speed camera (3) are connected to the main controller (1); the display screen (2) can rotate in a three-dimensional space, and the shooting position and angle of the camera (3) can be changed correspondingly.
Example two:
referring to fig. 2, the human-computer interaction system based on gesture control non-touch screen of the present invention comprises a main controller (1) and a display screen (2), and is characterized in that: the display screen (2) is horizontally placed on a desktop; a high-speed camera (3) vertical to the display screen (2) is arranged right above the display screen (2); a polarizing plate (4) is arranged in front of the camera (3); eliminating polarized light on the display screen (2) by adjusting the rotation angle of the polaroid (4); a display screen (2) and a high-speed camera (3) are connected to a main controller (1).
Example three:
referring to fig. 1, 2 and 3, the human-computer interaction method for controlling a non-touch screen based on gestures according to the present invention is implemented by using the system described in fig. 1 and 2, eliminating light on a display screen by using a polarizer, and controlling the operation of software through the change of gestures, and is characterized in that the implementation process is as follows:
1. starting a human-computer interaction system;
2. setting a working range in a human-computer interaction system;
3. recognizing the gesture;
4. selecting different gestures according to different software functions to control the software;
5. the system is shut down.
Example four:
the present embodiment is basically the same as the third embodiment, and the features are as follows:
in the third embodiment, the step 1 starts a human-computer interaction system: the high-speed camera switch and the display screen switch are turned on, then the main controller is started to control the operation of the high-speed camera and the display screen, and light emitted by the display screen is filtered out through the polaroid.
In the third embodiment, the step 2 sets a working range in the human-computer interaction system: and (3) defining the display screen as a background object, taking only the range of the space position of the display screen in the shot image as a man-machine interaction working range, and neglecting objects outside the range.
Example five:
referring to fig. 1, 2 and 4, the human-computer interaction method for controlling a non-touch screen based on gestures according to the present invention is operated by using the system shown in fig. 1 and 2, eliminates light on a display screen by using a polarizer, and controls software operation based on gesture recognition, wherein the gesture recognition method is characterized by comprising the following implementation processes:
1. obtaining an image of a foreground human body;
2. obtaining the shape of the hand;
3. converting a video image containing hand information into a binary image;
4. extracting the outer contour of a hand in the image;
5. extracting a palm center point;
6. extracting the outline of the palm;
7. extracting palm contour characteristic points;
8. acquiring palm contour feature points in a database;
9. and recognizing the gesture by using a shape space theory method.
Example six:
this embodiment is substantially the same as the fifth embodiment, and is characterized in that:
in the fifth embodiment, the step 1 obtains an image of a foreground human body: a user places a handle in a working range area of a human-computer interaction system, and obtains a video image containing the handle through a high-speed camera; the light emitted by the display screen is removed by adjusting the rotation angle of the polaroid sheet placed in front of the camera, and an image of the foreground human body is obtained.
Example five step 2 described above yields the shape of the hand: the color of the image is converted from the traditional RGB format to the HSV format, so that the difference between the skin color and other colors can be increased, the human body information containing the skin is extracted, the information contains the image information of the hand, partial noise is removed by using a Gaussian denoising method and a mathematical morphology method, and correct information in the hand image is reserved.
In the fifth embodiment, the step 3 converts the video image containing the hand information into a binary image.
In the fifth embodiment, the step 4 extracts the hand outline in the image: the object in the image contains an inner contour and an outer contour, where only the outer contour is extracted and the inner contour is eliminated; the area of each outline is calculated, and if the area is too small, the outline is deleted as noise, so that the hand outline is kept in the whole image.
Example five the step 5 described extracts the palm centre point: the hand contour of the user may contain partial arm contour information, which may affect the recognition of gestures; the shortest distance from the palm center point to the outline is the longest in the shortest distances from the points on the center lines of all the shapes to the outline, so the palm center point can be found by utilizing the property; the formula for obtaining the palm center point is as follows:
p hand shape interior arbitrary points, psIs the point on the outer contour of the hand, | | p, psI means p and psEuclidean distance between; the central point p of the palm can be obtained by the formulac；
In example five step 6 extracts the outer contour of the palm: the shortest distance formula from the palm center point to the hand outer contour is as follows:
ds＝||pc-ps||，
pcis the palm center point, p, calculated by equation (1)sIs a point on the outer contour of the hand, dsIs the shortest distance from the center point of the palm to the outer contour of the hand; the hand shape is divided by drawing a circle by taking the length of twice the shortest distance as a radius and the center point of the palm as the center of the circle, so that the shape of the arm part can be cut off; sometimes, part of the shape of the finger is also cut away; because the shape of the arm extends to the outside of the working area in the working area, the shape which does not extend to the outside of the working area outside the circle is used as the finger shape, so that the finger shape is recovered to obtain the complete palm shape; if all connected shapes outside the circle do not extend outside the work area, the circle outer shape area is used to determine if it is part of the palm, and the formula is as follows:
wherein S isiIs the circle outline area, S, of the shape of the connecting circle inner handcIs the area of the circle, ζ is the threshold value, setting
The shape of i satisfying the above formulaConsider a portion of the palm.
In the fifth embodiment, step 7 extracts the feature points of the palm contour: taking one of two cutting points of a palm and an arm as an initial characteristic point, taking the other one as an end characteristic point, starting from the initial characteristic point, taking N points along a contour line to the end characteristic point with equal arc length, taking M points from the end characteristic point to the initial characteristic point with equal arc length in the opposite direction, wherein the characteristic points on the contour line are N + M + 2; the more the feature points are, the more accurate the gesture recognition is; conversely, the fewer the feature points, the faster the speed; the specific number is determined by the user.
In the fifth embodiment, step 8 is to obtain the feature points of the palm contour in the database: step 2 to step 7, the hand image in the database is processed to obtain the characteristic points of the palm outline of the database; if the hand image in the database is the palm outline image after 2 to 6 steps, the system only needs to adopt 7 steps according to the requirement of extracting the number of the feature points on the contour line in each operation.
In the fifth embodiment, the step 9 uses a shape space theory method to recognize the gesture: a plurality of new shapes formed by scaling of one shape at different positions, rotation directions and different scales can be represented by one point in a shape space, and before gesture recognition is carried out by utilizing a shape space theory, palm contour feature points need to be projected into the shape space; two different shape similarities can be represented by Procrustean distance in shape space, which is expressed as follows in real number space:
dp[O(τ1),O(τ2)]＝inf[cos-1(＜x,y＞):x∈O(τ1),y∈O(τ2)]，
wherein, tau1And τ2Respectively setting two palm outline feature point sets which need to be compared with similarity; tau is1And τ2Projecting the two vectors into a hypersphere, and marking the hypersphere as Pre-shape space; o (tau)1) And O (τ)2) Are each tau1And τ2A great circle in the Pre-shape space; < x, y > is the inner product between x and y;
in the case of complex space, Procrumean distance formula is as follows:
wherein, tau1jAnd τ2jAre each tau1And τ2The jth complex coordinate of (a); tau is*Is the complex conjugate of τ;
by the above formula, the similarity of each standard hand and the observing hand in the database is calculated if their distance satisfies the following formula:
dp[O(τ1),O(τ2)]≤ξ,
xi is a threshold value for comparing similarity of a standard hand and an observation hand, and may have different values according to different numbers of contour points, and xi is set to be 0.4;
the standard hand profiles meeting the requirements can be many, and the standard hand with the shortest Procrusted distance between the standard hand profiles and the observation hand is used as the identification hand type of the observation hand;
if all the standard hands are traversed, the hand type of the observation hand is not recognized, and then the user is required to adjust the hand posture to inform that the recognition is failed; in addition, by this step, the shape of the face in the video can be eliminated.
Claims (4)
1. The human-computer interaction system based on the gesture control non-touch screen comprises a main controller (1) and a display screen (2), and is characterized in that: the display screen (2) is vertically arranged; a high-speed camera (3) is arranged above the display screen (2) in an inclined way and faces the display screen (2); a polarizing plate (4) is arranged in front of the high-speed camera (3); eliminating polarized light on the display screen (2) by adjusting the rotation angle of the polaroid (4); the display screen (2) and the high-speed camera (3) are connected to the main controller (1); the display screen (2) can rotate in a three-dimensional space, and the high-speed camera (3) correspondingly changes the shooting position and angle;
the human-computer interaction system based on the gesture control non-touch screen is operated by adopting the following human-computer interaction method based on the gesture control non-touch screen, and the operation steps are as follows:
1) starting a human-computer interaction system;
2) setting a working range in a human-computer interaction system;
3) recognizing the gesture;
4) selecting different gestures according to different software functions to control the software;
5) shutting down the system;
in the gesture recognition method in step 3), the adopted gesture recognition method is a method based on a shape space theory, and is specifically described as follows:
(6-1) obtaining an image of the foreground human body:
a user places a handle in a working range area of the man-machine interaction system, and obtains a video image containing the handle through a high-speed camera (3); removing the light emitted by the display screen (2) by adjusting the rotation angle of a polaroid (4) placed in front of the high-speed camera (3) to obtain an image of a foreground human body;
(6-2) obtaining the shape of the hand:
the color of the image of the human body is converted from an RGB format to an HSV format, so that the difference between the skin color and other colors can be increased, the human body information containing the skin is extracted, the human body information contains the image information of the hand, partial noise is removed by using a Gaussian denoising method and a mathematical morphology method, and correct information in the hand image is reserved;
(6-3) converting the video image containing the hand information into a binary image;
(6-4) extracting the hand outline in the binary image containing the hand information:
in the binary image containing hand information, the object contains an inner contour and an outer contour, wherein only the outer contour is extracted and the inner contour is eliminated; calculating the area of each outline, and if the area is too small, deleting the outline as noise, so that the hand outline is reserved in the whole binary image containing the hand information;
(6-5) extracting the palm center point:
the hand contour of the user comprises partial arm contour information, which can influence the gesture recognition; the shortest distance from the palm center point to the outline is the longest in the shortest distances from the points on the center lines of all the shapes to the outline, so the palm center point can be found by utilizing the property; the formula for obtaining the palm center point is as follows:
p hand shape interior arbitrary points, psIs the point on the outer contour of the hand, | | p, psI means p and psEuclidean distance between; the central point p of the palm can be obtained by the formulac；
(6-6) extracting the external contour of the palm:
the shortest distance formula from the palm center point to the hand outer contour is as follows:
ds＝||pc-ps||， (2)
pcis the palm center point, p, calculated by equation (1)sIs a point on the outer contour of the hand, dsIs the shortest distance from the center point of the palm to the outer contour of the hand; the hand shape is divided by drawing a circle by taking the length of twice the shortest distance as a radius and the center point of the palm as the center of the circle, so that the shape of the arm part can be cut off; sometimes, part of the shape of the finger is also cut away; because the shape of the arm extends to the outside of the working area in the working area, the shape which does not extend to the outside of the working area outside the circle is used as the finger shape, so that the finger shape is recovered to obtain the complete palm shape; if all connected shapes outside the circle do not extend outside the work area, the circle outer shape area is used to determine if it is part of the palm, and the formula is as follows:
wherein S isiIs the circle outline area, S, of the shape of the connecting circle inner handcIs the area of the circle, ζ is the threshold value, setting
The shape of i satisfying the formula (3) can be confirmed as a part of the palm;
(6-7) extracting palm contour characteristic points:
taking one of two cutting points of a palm and an arm as an initial characteristic point, taking the other one as an end characteristic point, starting from the initial characteristic point, taking N points along a contour line to the end characteristic point with equal arc length, taking M points from the end characteristic point to the initial characteristic point with equal arc length in the opposite direction, wherein the characteristic points on the contour line are N + M + 2;
(6-8) acquiring palm contour characteristic points in the database:
processing the hand image in the database to obtain the characteristic points of the palm outline of the database in the steps (6-2) to (6-7); if the hand image in the database is the palm outline image after the steps from (6-2) to (6-6), the system only needs to adopt the step (6-7) according to the requirement of extracting the number of the feature points on the contour line during each operation;
(6-9) recognizing the gesture by using a shape space theory method:
a plurality of new shapes formed by scaling of one shape at different positions, rotation directions and different scales can be represented by one point in a shape space, and before gesture recognition is carried out by utilizing a shape space theory, palm contour feature points need to be projected into the shape space; two different shape similarities can be represented by Procrustean distance in shape space, which is expressed as follows in real number space:
dp[O(τ1),O(τ2)]＝inf[cos-1(＜x,y＞):x∈O(τ1),y∈O(τ2)]， (4)
wherein, tau1And τ2Respectively setting two palm outline feature point sets which need to be compared with similarity; tau is1And τ2Projecting the two vectors into a hypersphere, and marking the hypersphere as Pre-shape space; o (tau)1) And O (τ)2) Are each tau1And τ2A great circle in the Pre-shape space; < x, y > is the inner product between x and y;
in the case of complex space, Procrumean distance formula is as follows:
wherein, tau1jAnd τ2jAre each tau1And τ2The jth complex coordinate of (a); tau is*Is the complex conjugate of τ;
by formulas (4) and (5), the similarity of each standard hand and the observing hand in the database is calculated if their distance satisfies the following formula:
dp[O(τ1),O(τ2)]≤ξ, (6)
xi is a threshold value for comparing similarity of the standard hand and the observation hand, and xi is set to be 0.4;
among the standard hands satisfying the formula (6), the standard hand with the shortest Procrumean distance to the observation hand is taken as the identification hand type of the observation hand;
if all the standard hands are traversed, the hand type of the observation hand is not recognized, and then the user is required to adjust the hand posture to inform that the recognition is failed; in addition, by this step, the shape of the face in the video can be eliminated.
2. The human-computer interaction system based on the gesture control non-touch screen comprises a main controller (1) and a display screen (2), and is characterized in that: the display screen (2) is horizontally placed on a desktop; a high-speed camera (3) vertical to the display screen (2) is arranged right above the display screen (2); a polarizing plate (4) is arranged in front of the high-speed camera (3); eliminating polarized light on the display screen (2) by adjusting the rotation angle of the polaroid (4); the display screen (2) and the high-speed camera (3) are connected to the main controller (1);
the human-computer interaction system based on the gesture control non-touch screen is operated by adopting the following human-computer interaction method based on the gesture control non-touch screen, and the operation steps are as follows:
1) starting a human-computer interaction system;
2) setting a working range in a human-computer interaction system;
3) recognizing the gesture;
4) selecting different gestures according to different software functions to control the software;
5) shutting down the system;
in the gesture recognition method in step 3), the adopted gesture recognition method is a method based on a shape space theory, and is specifically described as follows:
(6-1) obtaining an image of the foreground human body:
a user places a handle in a working range area of the man-machine interaction system, and obtains a video image containing the handle through a high-speed camera (3); removing the light emitted by the display screen (2) by adjusting the rotation angle of a polaroid (4) placed in front of the high-speed camera (3) to obtain an image of a foreground human body;
(6-2) obtaining the shape of the hand:
the color of the image of the human body is converted from an RGB format to an HSV format, so that the difference between the skin color and other colors can be increased, the human body information containing the skin is extracted, the human body information contains the image information of the hand, partial noise is removed by using a Gaussian denoising method and a mathematical morphology method, and correct information in the hand image is reserved;
(6-3) converting the video image containing the hand information into a binary image;
(6-4) extracting the hand outline in the binary image containing the hand information:
in the binary image containing hand information, the object contains an inner contour and an outer contour, wherein only the outer contour is extracted and the inner contour is eliminated; calculating the area of each outline, and if the area is too small, deleting the outline as noise, so that the hand outline is reserved in the whole binary image containing the hand information;
(6-5) extracting the palm center point:
the hand contour of the user comprises partial arm contour information, which can influence the gesture recognition; the shortest distance from the palm center point to the outline is the longest in the shortest distances from the points on the center lines of all the shapes to the outline, so the palm center point can be found by utilizing the property; the formula for obtaining the palm center point is as follows:
p hand shape interior arbitrary points, psIs the point on the outer contour of the hand, | | p, psI means p and psEuclidean distance between; the central point p of the palm can be obtained by the formulac；
(6-6) extracting the external contour of the palm:
the shortest distance formula from the palm center point to the hand outer contour is as follows:
ds＝||pc-ps||， (2)
pcis the palm center point, p, calculated by equation (1)sIs a point on the outer contour of the hand, dsIs the shortest distance from the center point of the palm to the outer contour of the hand; the hand shape is divided by drawing a circle by taking the length of twice the shortest distance as a radius and the center point of the palm as the center of the circle, so that the shape of the arm part can be cut off; sometimes, part of the shape of the finger is also cut away; because the shape of the arm extends to the outside of the working area in the working area, the shape which does not extend to the outside of the working area outside the circle is used as the finger shape, so that the finger shape is recovered to obtain the complete palm shape; if all connected shapes outside the circle do not extend outside the work area, the circle outer shape area is used to determine if it is part of the palm, and the formula is as follows:
wherein S isiIs the circle outline area, S, of the shape of the connecting circle inner handcIs the area of the circle, ζ is the threshold value, setting
The shape of i satisfying the formula (3) can be confirmed as a part of the palm;
(6-7) extracting palm contour characteristic points:
taking one of two cutting points of a palm and an arm as an initial characteristic point, taking the other one as an end characteristic point, starting from the initial characteristic point, taking N points along a contour line to the end characteristic point with equal arc length, taking M points from the end characteristic point to the initial characteristic point with equal arc length in the opposite direction, wherein the characteristic points on the contour line are N + M + 2;
(6-8) acquiring palm contour characteristic points in the database:
processing the hand image in the database to obtain the characteristic points of the palm outline of the database in the steps (6-2) to (6-7); if the hand image in the database is the palm outline image after the steps from (6-2) to (6-6), the system only needs to adopt the step (6-7) according to the requirement of extracting the number of the feature points on the contour line during each operation;
(6-9) recognizing the gesture by using a shape space theory method:
a plurality of new shapes formed by scaling of one shape at different positions, rotation directions and different scales can be represented by one point in a shape space, and before gesture recognition is carried out by utilizing a shape space theory, palm contour feature points need to be projected into the shape space; two different shape similarities can be represented by Procrustean distance in shape space, which is expressed as follows in real number space:
dp[O(τ1),O(τ2)]＝inf[cos-1(＜x,y＞):x∈O(τ1),y∈O(τ2)]， (4)
wherein, tau1And τ2Respectively setting two palm outline feature point sets which need to be compared with similarity; tau is1And τ2Projecting the two vectors into a hypersphere, and marking the hypersphere as Pre-shape space; o (tau)1) And O (τ)2) Are each tau1And τ2A great circle in the Pre-shape space; < x, y > is the inner product between x and y;
in the case of complex space, Procrumean distance formula is as follows:
wherein, tau1jAnd τ2jAre each tau1And τ2The jth complex coordinate of (a); tau is*Is the complex conjugate of τ;
by formulas (4) and (5), the similarity of each standard hand and the observing hand in the database is calculated if their distance satisfies the following formula:
dp[O(τ1),O(τ2)]≤ξ, (6)
xi is a threshold value for comparing similarity of the standard hand and the observation hand, and xi is set to be 0.4;
among the standard hands satisfying the formula (6), the standard hand with the shortest Procrumean distance to the observation hand is taken as the identification hand type of the observation hand;
if all the standard hands are traversed, the hand type of the observation hand is not recognized, and then the user is required to adjust the hand posture to inform that the recognition is failed; in addition, by this step, the shape of the face in the video can be eliminated.
3. The human-computer interaction system for controlling the non-touch screen based on the gestures as claimed in claim 1 or 2, wherein: the step 1) starts a human-computer interaction system: firstly, a switch of the high-speed camera (3) and a switch of the display screen (2) are turned on, then the main controller (1) is started to control the operation of the high-speed camera (3) and the display screen (2), and light emitted by the display screen (2) is filtered out through the polaroid sheet (4).
4. The human-computer interaction system based on the gesture control non-touch screen as claimed in claim 1 or 2, wherein: and 2) setting a working range in the human-computer interaction system: the display screen (2) is specified as a background object, only the range of the space position of the display screen (2) is taken as a man-machine interaction working range in a shot image, and objects outside the range are ignored.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710414437.3A CN107272893B (en) | 2017-06-05 | 2017-06-05 | Man-machine interaction system and method for controlling non-touch screen based on gestures |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710414437.3A CN107272893B (en) | 2017-06-05 | 2017-06-05 | Man-machine interaction system and method for controlling non-touch screen based on gestures |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107272893A CN107272893A (en) | 2017-10-20 |
| CN107272893B true CN107272893B (en) | 2020-09-08 |
Family
ID=60064966
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710414437.3A Active CN107272893B (en) | 2017-06-05 | 2017-06-05 | Man-machine interaction system and method for controlling non-touch screen based on gestures |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107272893B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109409277B (en) * | 2018-10-18 | 2020-11-24 | 北京旷视科技有限公司 | Gesture recognition method and device, intelligent terminal and computer storage medium |
| CN113741701A (en) * | 2021-09-30 | 2021-12-03 | 之江实验室 | Brain nerve fiber bundle visualization method and system based on somatosensory gesture control |
| CN115562501B (en) * | 2022-12-05 | 2023-03-03 | 南京达斯琪数字科技有限公司 | Man-machine interaction method for rotary scanning display |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8433138B2 (en) * | 2008-10-29 | 2013-04-30 | Nokia Corporation | Interaction using touch and non-touch gestures |
| CN102044156B (en) * | 2009-10-13 | 2012-11-14 | 上海高德威智能交通系统有限公司 | Polarization correction device and method |
| CN102004913A (en) * | 2010-06-25 | 2011-04-06 | 南通棉花机械有限公司 | Method for detecting transparent plastic impurities in seed cotton |
| CN102663364A (en) * | 2012-04-10 | 2012-09-12 | 四川长虹电器股份有限公司 | Imitated 3D gesture recognition system and method |
| CN104331158B (en) * | 2014-10-29 | 2018-05-25 | 山东大学 | The man-machine interaction method and device of a kind of gesture control |
| CN104750250A (en) * | 2015-03-04 | 2015-07-01 | 深圳市祈锦通信技术有限公司 | Gesture-interaction-supported projection method and system |
| CN106200971A (en) * | 2016-07-07 | 2016-12-07 | 广东技术师范学院 | Man-machine interactive system device based on gesture identification and operational approach |
-
2017
- 2017-06-05 CN CN201710414437.3A patent/CN107272893B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107272893A (en) | 2017-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10394334B2 (en) | Gesture-based control system | |
| Li | Hand gesture recognition using Kinect | |
| Yang et al. | Dynamic hand gesture recognition using hidden Markov models | |
| CN109597485B (en) | Gesture interaction system based on double-fingered-area features and working method thereof | |
| Pan et al. | Real-time sign language recognition in complex background scene based on a hierarchical clustering classification method | |
| CN107688391A (en) | A kind of gesture identification method and device based on monocular vision | |
| CN107272893B (en) | Man-machine interaction system and method for controlling non-touch screen based on gestures | |
| CN104866824A (en) | Manual alphabet identification method based on Leap Motion | |
| CN104821010A (en) | Binocular-vision-based real-time extraction method and system for three-dimensional hand information | |
| CN110531853B (en) | Electronic book reader control method and system based on human eye fixation point detection | |
| Wu et al. | Vision-based fingertip tracking utilizing curvature points clustering and hash model representation | |
| CN104038799A (en) | Three-dimensional television-oriented gesture manipulation method | |
| CN103092437A (en) | Portable touch interactive system based on image processing technology | |
| Jaemin et al. | A robust gesture recognition based on depth data | |
| CN107368181B (en) | Gesture recognition method and device | |
| Francis et al. | Significance of hand gesture recognition systems in vehicular automation-a survey | |
| WO2013078657A1 (en) | A gesture recognition method, an apparatus and a computer program for the same | |
| Karabasi et al. | A model for real-time recognition and textual representation of malaysian sign language through image processing | |
| Chen et al. | Depth-based hand gesture recognition using hand movements and defects | |
| Lee et al. | A Hand gesture recognition system based on difference image entropy | |
| Bakar et al. | Real-time rotation invariant hand tracking using 3D data | |
| Kakkoth et al. | Survey on real time hand gesture recognition | |
| Lan et al. | Data fusion-based real-time hand gesture recognition with Kinect V2 | |
| Abdallah et al. | An overview of gesture recognition | |
| CN115713808A (en) | Gesture recognition system based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |