CN108549489B

CN108549489B - gesture control method and system based on hand shape, posture, position and motion characteristics

Info

Publication number: CN108549489B
Application number: CN201810392063.4A
Authority: CN
Inventors: 刘春燕; 孙晅; 李美娟
Original assignee: Harbin Top Technology Co Ltd
Current assignee: Harbin Top Technology Co Ltd
Priority date: 2018-04-27
Filing date: 2018-04-27
Publication date: 2019-12-13
Anticipated expiration: 2038-04-27
Also published as: CN108549489A

Abstract

the invention provides a gesture control method and a gesture control system based on hand shape, posture, position and motion characteristics, which can be used for selecting and controlling a plurality of groups of equipment and a plurality of functional modules in the equipment. According to the method, the gesture action can be captured through the camera, the shape, the posture, the position and the motion characteristics are extracted, and on the basis, the characteristics are analyzed to identify the gesture of the user, so that the selection and the control of equipment or a function module are realized.

Description

gesture control method and system based on hand shape, posture, position and motion characteristics

Technical Field

The invention relates to a gesture control method and system based on hand shape, posture, position and motion characteristics, and belongs to the technical field of gesture recognition.

background

the gesture control has the characteristics of non-contact and convenience in operation, and is one of important research directions in a man-machine interaction mode. In the common gesture control methods at present, the electromagnetic induction method is difficult to identify the fine movement of fingers; the control method based on the wearable device requires the user to additionally wear a specific device, which increases the burden on the user. In the image-based gesture control method, only the number of fingers or the two-dimensional planar motion track of the hand is recognized, and the information (such as the direction, the three-dimensional motion track and the like) contained in the image is not completely mined; or only adopt the operation mode similar to the traditional mouse or touch screen such as sliding, clicking and the like. Therefore, the existing gesture control method is difficult to control a complex system comprising multiple groups of equipment or having multi-level functions in an interactive mode or only by using a single simple action; or imitate the traditional mouse or touch-sensitive screen operation, do not change the man-machine interaction mode from the essence. In addition, part of the methods depend on the traditional human-computer interaction interface, a user needs to check the display equipment in real time, blind operation cannot be realized, and the method has obvious limitation in practical application. For example, during driving of a car, viewing additional display devices may result in distractions to the driver, creating a safety hazard.

Disclosure of Invention

The invention provides a gesture control method based on hand shape, posture, position and motion characteristics, aiming at solving the problem that the gesture control method in the prior art can not be used for selecting and controlling a plurality of groups of equipment and a plurality of functional modules in the equipment, and the adopted technical scheme is as follows:

The gesture control method comprises the following steps:

reading image data of an input image;

step two, extracting hand features in the image data in the step one, and obtaining a hand feature result; the hand features comprise hand form features, hand posture features, hand position features and hand action features; the hand characteristic results comprise hand form characteristic results, hand posture characteristic results, hand position characteristic results and hand action characteristic results;

thirdly, fusing, analyzing and gesture recognizing the hand characteristic data to obtain a gesture recognition result;

step four, storing the gesture recognition result of the step three;

step five, judging whether the gesture recognition result in the step four is a complete gesture, and if the gesture recognition result is the complete gesture, sending a gesture command corresponding to the gesture recognition result; if not, iteratively executing the first step to the fourth step until a complete gesture is recognized.

Further, the hand features are described in a world coordinate system and a hand coordinate system; the world coordinate system o-xyz is taken as an absolute coordinate system of the whole system; when the hand coordinate system o ' -x ' y ' z ' is unfolded by stretching the palm, the intersection point of the straight line of the thumb and the straight line of the middle finger is the original point, the straight line of the middle finger is the x ' axis, and the direction pointed by the finger tip is the positive direction; the direction vertical to the palm is the y' axis direction, when the right hand is used as the control hand, the palm direction is the forward direction, and when the left hand is used, the back direction is the forward direction; the z ' axis is perpendicular to the plane formed by the x ' and y ' axes, and the thumb direction is the positive direction.

Further, the hand morphological feature of the second step is extracted by an image fractal method, and the extraction process of the image fractal method includes:

step1, reading image data of an input image;

step2, carrying out threshold segmentation by using the image gray scale or color information of the image data in the step1 to obtain a hand region;

step3, extracting the edge contour of the hand area in the step2 to obtain a hand edge contour result;

Step4, analyzing the hand edge contour result obtained in the step3, and judging the number and the pointing direction of fingers through the bulges and the depressions of the edge contour curve in the edge contour result to obtain a hand morphological feature result;

And 5, outputting the hand form feature result in the step 4.

further, the hand form features of the second step are extracted by a template matching method, and the extraction process of the template matching method comprises the following steps:

firstly, reading image data of an input image, and loading preset template data;

secondly, carrying out full-image matching on the image by adopting a relevant filtering method on a preset template based on gray scale or color characteristics in the preset template in the first step; carrying out full-image matching on the image by utilizing a cascade classifier on a preset template based on texture feature characteristics of HOG, Haar or LPB in the preset template in the first step to obtain a matching result;

And thirdly, determining hand form characteristics according to the matching result in the second step, and outputting the hand form characteristics.

further, the step two of extracting the hand gesture features comprises the following steps:

a, reading the hand form feature result, and loading the hand form feature result into a hand preset three-dimensional model;

b, extracting feature points by using the hand form feature result in the step a;

C, matching the feature point positions and the pixel values in the step b with the preset three-dimensional model to obtain a preset three-dimensional model matching result;

d, establishing a hand coordinate system according to the preset three-dimensional model matching result in the step c;

E, solving hand gesture parameters by using a PnP method according to the hand coordinate system in the step d, wherein the hand gesture parameters are the hand gesture characteristics;

and d, outputting a characteristic result of the hand posture characteristic.

further, the hand position features in the step two are extracted by using a hand posture feature result, and the extraction process of the hand position features comprises the following steps:

step I, reading the hand gesture feature result

Step II, calculating hand position parameters by using the hand posture characteristic result in the step I through a world coordinate system and a hand coordinate system, wherein the hand position parameters are the hand position characteristics;

And III, outputting the hand position feature result in the step II.

Further, the hand position features in the step two are extracted by using hand form feature results, and the extraction process of the hand position features comprises the following steps:

step A, reading the hand form feature result;

b, calculating hand position parameters by using the hand form characteristic results in the step A and through a hand centroid and a world coordinate system, wherein the hand position parameters are the hand position characteristics;

and C, outputting the hand position feature result in the step B.

further, the step two of extracting the hand motion features comprises the following steps:

step1, reading hand form characteristic results, hand posture characteristic results and hand position characteristic results of a preorder image and a current image;

step2, performing difference analysis on the hand form characteristic result, the hand posture characteristic result and the hand position characteristic result in the step1, and obtaining hand movements after the difference analysis; the hand action is a hand action characteristic;

Step3, outputting a hand action characteristic result;

And step4, storing the hand motion characteristic result in the step3 so as to be convenient for the subsequent extraction process to call.

Further, extracting the hand features in the second step by using a depth image feature extraction method, wherein the extraction process comprises the following steps:

step 1: reading image data of an input image, and loading a preset three-dimensional skeleton model;

Step 2: segmenting a hand region input in the image using depth information;

Step 3: extracting hand feature points in the hand region;

step 4: performing model matching by using the hand characteristic points of step3 to obtain a model matching result;

Step 5: establishing a hand coordinate system;

Step 6: extracting hand morphological features, hand posture features and hand position features according to the model matching result of step4 and the hand coordinate system of step 5;

Step 7: and calculating the hand motion characteristics according to the difference of the front frame image and the rear frame image of the input image.

A gesture control system for realizing the gesture control method of claim 1 adopts the following technical scheme:

The control system comprises a main controller, a data acquisition module, a data processing module, an instruction output module and an operation feedback module; the data acquisition signal control interaction end of the main controller is connected with the data control interaction end of the data acquisition module; the data processing signal control interaction end of the main controller is connected with the data control interaction end of the data processing module; the command output signal control interaction end of the main controller is connected with the data control interaction end of the command output module; an operation feedback signal control interaction end of the main controller is connected with a data control interaction end of the operation feedback module; the data output end of the data acquisition module is connected with the data input end of the data processing module; and the data output end of the data processing module is connected with the data input end of the instruction output module.

The invention provides a gesture control method based on hand shape, posture, position and motion characteristics, which can be used for selecting and controlling multiple groups of equipment and multiple functional modules in the equipment. The method can capture gesture actions through the camera, extract form, posture, position and motion characteristics, analyze the characteristics on the basis to recognize the gestures of a user, further realize the selection and control of equipment or a function module, and control the regulation rate of a function state through the action amplitude of the gestures. The user only needs to make simple gesture actions in a small range to complete control, and can define preset gestures according to actual requirements, so that the use is convenient.

the invention also provides a control system corresponding to the method, which can capture and identify the gesture actions so as to control a plurality of groups of equipment and a plurality of functional modules in the equipment. The control system also has a feedback function, so that a user can obtain real-time light, sound or vibration feedback when performing gesture actions, and the control system is simple to operate and good in interactivity.

the invention has the beneficial effects that:

1. The gesture recognition method can realize gesture recognition based on the monocular two-dimensional image or the depth image, and has wider application range compared with the current mainstream method only based on the depth camera.

2. The control system provided by the invention can adopt a monocular camera as a gesture motion capturing device.

3. The invention can realize the selection and control of a plurality of groups of equipment and a plurality of functional modules in the equipment through a single coherent gesture.

4. When the invention is used for adjusting the functional state with a certain range interval, the speed of adjustment can be controlled through the gesture action amplitude.

5. The gesture control method and the gesture control system provided by the invention have the advantages that the controllable equipment quantity is large, and the controllable function hierarchy is complex in control effect.

6. the gesture control method and the gesture control system provided by the invention are obviously different from the operation modes of the traditional mouse and the touch screen in the interaction mode, but do not simulate the operation modes.

7. the gesture control method provided by the invention can realize blind operation.

drawings

FIG. 1 is a general flowchart of a gesture control method according to the present invention.

Fig. 2 is a schematic view of the world coordinate system and the hand coordinate system according to the present invention.

FIG. 3 is an exemplary diagram of hand gesture features of the present invention.

FIG. 4 is an exemplary diagram of a hand position feature of the present invention.

FIG. 5 is an exemplary diagram of a hand movement feature according to the present invention.

Fig. 6 is a flowchart of an image fractal method according to the present invention.

fig. 7 is a flowchart of the template matching method according to the present invention.

fig. 8 is a flow chart of hand gesture feature extraction according to the present invention.

Fig. 9 is a flowchart of the hand position feature extraction according to the present invention.

Fig. 10 is a flow chart of hand motion feature extraction according to the present invention.

Fig. 11 is a flow chart of feature extraction of a depth image according to the present invention.

FIG. 12 is a diagram of a gesture control system according to the present invention.

fig. 13 is a diagram showing a hardware installation manner of the central control system of the automobile.

FIG. 14 is a schematic view of a driver side window opening gesture.

FIG. 15 is a schematic view of a close driver side window gesture.

FIG. 16 is a schematic view of the open and close modes of the sunroof.

FIG. 17 is a diagram illustrating a translating skylight opening gesture.

FIG. 18 is a diagram illustrating a translating sunroof closing gesture.

FIG. 19 is a schematic view of a tilt open skylight gesture.

FIG. 20 is a schematic view of a tilt close skylight gesture.

fig. 21 is a schematic diagram of installation of a smart home central control device.

Fig. 22 is a diagram illustrating a floor lamp turning on gesture.

fig. 23 is a schematic diagram illustrating a turn-off gesture of the floor lamp.

Fig. 24 is a diagram illustrating a decreasing brightness gesture of the floor lamp.

Fig. 25 is a diagram illustrating a gesture of increasing the brightness of the floor lamp.

FIG. 26 is a diagram illustrating a multimedia device opening gesture.

FIG. 27 is a diagram illustrating a close gesture for a multimedia device.

FIG. 28 is a diagram illustrating playing and pausing of a multimedia device.

FIG. 29 is a schematic diagram illustrating a gesture for switching the multimedia device to a previous file.

FIG. 30 is a diagram illustrating a fast-rewinding gesture of a current file of a multimedia device.

FIG. 31 is a diagram illustrating a multimedia device switching to a next file gesture.

FIG. 32 is a diagram illustrating a fast forward gesture for a current file of a multimedia device.

FIG. 33 is a diagram illustrating a volume-down gesture of the multimedia device.

FIG. 34 is a diagram illustrating a volume up gesture of the multimedia device.

FIG. 35 is a flow chart showing the relationship between the convex and concave curves and the number and direction of fingers.

FIG. 36 is a diagram illustrating an example of a gesture for determining relationship between protrusions and depressions of a curve and the number and directions of fingers.

FIG. 37 is a diagram of an example of a second gesture for determining the relationship between the protrusions and depressions of the curve and the number and directions of the fingers.

Detailed Description

the present invention will be further described with reference to the following specific examples, but the present invention is not limited to these examples.

Example 1:

the invention provides a gesture control method based on hand form, posture, position and motion characteristics. As shown in fig. 1, the gesture control method includes firstly reading an input image, sequentially extracting one or more of hand morphology, posture, position and motion characteristics from the image based on actual application requirements, fusing and analyzing the characteristics, sending a complete gesture command when the command is detected, and otherwise storing a current analysis result and performing processing analysis by combining with subsequent images, specifically:

reading image data of an input image;

step two, extracting hand features in the image data in the step one, and obtaining a hand feature result;

Step four, storing the gesture recognition result of the step three;

The hand features are described in a world coordinate system and a hand coordinate system; as shown in fig. 2, the world coordinate system o-xyz is taken as the absolute coordinate system of the entire system; the origin position and the three-axis direction are selected according to the actual system requirements. When the hand coordinate system o ' -x ' y ' z ' is unfolded by stretching the palm, the intersection point of the straight line of the thumb and the straight line of the middle finger is the original point, the straight line of the middle finger is the x ' axis, and the direction pointed by the finger tip is the positive direction; the direction vertical to the palm is the y' axis direction, when the right hand is used as the control hand, the palm direction is the forward direction, and when the left hand is used, the back direction is the forward direction; the z ' axis is perpendicular to the plane formed by the x ' and y ' axes, and the thumb direction is the positive direction.

The hand features comprise hand form features, hand posture features, hand position features and hand action features; the hand characteristic results comprise hand form characteristic results, hand posture characteristic results, hand position characteristic results and hand action characteristic results.

wherein, the hand shape characteristic refers to the external contour shape of the hand. When the user makes actions of extending fingers, making a fist or opening a palm, the gesture can be quickly identified. In this embodiment, different hand configurations may be used to implement the following functions:

(1) selecting a certain device from a plurality of devices, or selecting different function modules of the same device, wherein the device or the module responds to a subsequent gesture control instruction;

(2) adjusting different states of a specific function under the selected equipment or functional module, wherein if a finger is extended out to control the equipment to be opened, the equipment is controlled to be closed when a fist is clenched;

(3) And controlling the speed of the change of the functional state, if a certain functional state needs to be increased, when one finger is stretched out, the state is adjusted slowly, and when two fingers are stretched out, the state is adjusted quickly.

the hand posture characteristics include the azimuth direction formed by the pitching, swinging, rotating and the like of the hand, and can be represented by the included angle between the hand coordinate system and the world coordinate system, and the size of the included angle represents the posture amplitude, as shown by the included angle a between o 'x' and ox in fig. 3.

Different hand gestures in this embodiment can be used to implement the following functions:

(2) Adjusting different states of a specific function under the equipment or the functional module, such as controlling the equipment to be turned on and off by utilizing the up-down pitching gesture of the hand and controlling the equipment to be turned on and turned off, and controlling the function state of the equipment to be turned up and turned down by utilizing the left-right swinging gesture;

(3) The speed of the change of the functional state can be controlled according to the difference of the attitude amplitude, if a certain functional state needs to be increased, when the attitude amplitude is large, the state adjustment is fast, and when the attitude amplitude is small, the adjustment is slow.

the hand position feature indicates the spatial position coordinates of the hand, and can be represented by the coordinates of the hand coordinate system origin in the world coordinate system, and the distance between the hand coordinate system origin and the world coordinate system origin represents the displacement, as indicated by the line segment l in fig. 4. When the precision requirement is not high, the coordinates of the centroid of the hand in the world coordinate system and the distance from the origin can be directly calculated.

Different hand positions in this embodiment can be used to implement the following functions:

(3) and controlling the speed of the change of the functional state according to the displacement, if a certain functional state needs to be increased, when the displacement is large, the state adjustment is fast, and when the displacement is small, the adjustment is slow.

the hand motion characteristic refers to the change process of hand form, posture or position characteristic within a period of time. As shown in fig. 5, which gives a set of hand motion examples where the hand rotates about the oy axis while translating in the ox direction. In addition, hand movements may also include changes in form, such as extending or retracting fingers, etc.

Different hand movements in this embodiment can be used to implement the following functions:

(2) Adjusting different states of a specific function under the equipment or the functional module;

(3) and controlling the speed of the change of the functional state according to the speed and the amplitude of the action.

In this embodiment, hand features are extracted based on a monocular two-dimensional image method according to differences of input images, and a specific extraction method is as follows

And the hand morphological characteristics in the second step are extracted by an image fractal method or a template matching method or a method combining the image fractal method and the template matching method, wherein the flow of the image fractal method is shown in fig. 6, firstly, based on the gray scale or color information of the hand image, hand pixels are obtained by utilizing threshold segmentation, then, hand edge information is extracted, and the number and the pointing direction of the extending fingers can be judged through the bulges and the depressions of the edge curve. The extraction process of the image fractal method comprises the following steps:

Step1, reading image data of an input image;

and 5, outputting the hand form feature result in the step 4.

Wherein, the flow of judging the relationship between the convex and concave curves and the number and direction of the fingers in step4 is shown in fig. 35, specifically:

1. after the hand edge image is obtained, traversing all edge points in the image, and calculating the curvature;

2. Judging the type of the edge points according to the curvature value, wherein the edge points are convex points when the curvature value is larger than the convex judgment threshold value, the edge points are concave points when the curvature value is smaller than the concave judgment threshold value, and the rest are common edge points as shown in fig. 36 to 37 (for simplicity, the edges in the figure and the subsequent figures are omitted and not drawn); wherein, the curvature at the point A is larger than the bulge judgment threshold value and is marked as a bulge point; the curvatures of the point B and the point C are smaller than a sinking judgment threshold value and are marked as sinking points;

3. And traversing each convex point, and analyzing the position relation between the point and the adjacent concave point:

a) the point A and the point B are adjacent, the point A and the point C are adjacent, and no other convex points are arranged between the point B and the point C.

b) And connecting the point A with the point B and the point C respectively, wherein the lengths of the line segments AB and AC both meet the preset threshold value.

c) The size of an included angle formed by the AB and the AC meets a preset threshold value.

d) combining the three points a), b), c), consider point a to be a fingertip, and the finger pointing direction can be described by line segments AB and AC.

e) If the three points a), b), c) are not all satisfied, point A is not a fingertip.

The template matching method flow is as shown in fig. 7, firstly training based on hand gray scale, color or texture prior knowledge under different forms to obtain a series of characteristic templates, then carrying out full-image matching in an input image by utilizing the templates, and obtaining the best matching result which is the hand form in the current image. The extraction process of the module matching method comprises the following steps:

The hand posture characteristic extraction needs to be carried out on the basis of morphological characteristics. Firstly, a series of three-dimensional models are established based on different hand forms, feature points are extracted according to the obtained hand forms, image feature points are matched with a preset model according to feature Point positions and pixel values, a hand coordinate system in the current state is established, finally, gesture parameters of the hand in the world coordinate system are solved by utilizing a PnP method (Perspective-n-Point, multi-Point pose), as shown in FIG. 8, and the extraction process of the hand gesture features in the second step comprises the following steps:

And d, outputting a characteristic result of the hand posture characteristic.

the extraction of the hand position features needs to be carried out on the basis of the posture or morphological features. When extracting based on the posture characteristics, the obtained hand coordinate system and the world coordinate system can be directly used for calculation; when extracting based on morphological features, the centroid can be calculated by using the hand edge contour obtained above, and then the coordinates of the centroid in the world coordinate system and the distance from the origin are obtained, as shown in fig. 9, the specific flow is as follows:

When the hand position features in the step two are extracted by using the hand posture feature result, the extraction process of the hand position features comprises the following steps:

Step I, reading the hand gesture feature result

step II, calculating hand position parameters through a world coordinate system and a hand coordinate system by using the hand posture characteristic result obtained in the step I, wherein the hand position parameters are the hand position characteristics (in the hand posture characteristic extraction process, the hand posture characteristic extraction method is applied to a PnP algorithm which can output a rotation transformation matrix and a translation transformation matrix of the hand coordinate system relative to the world coordinate system, wherein the rotation transformation matrix describes respective included angles of the hand coordinate system relative to the world coordinate system along three coordinate axes of xyz, the group of included angles are the hand posture characteristics, the translation transformation matrix describes three-dimensional coordinates of an origin o' of the hand coordinate system in the world coordinate system o-xyz, and the group of coordinates are the hand position characteristics)

and III, outputting the hand position feature result in the step II. When the hand position features in the step two are extracted by using the hand form feature result, the extraction process of the hand position features comprises the following steps:

step A, reading the hand form feature result;

and C, outputting the hand position feature result in the step B.

the calculation process of calculating the hand position characteristics by combining the world coordinates and the hand centroids in the step B is as follows:

1. in the process of establishing the world coordinate system, the mapping relation of the image coordinates of the image captured by the camera to the world coordinates is included.

2. After the hand edge contour is obtained through extraction, the average value of the image coordinates of all points on the edge contour is calculated, and the average value is the image coordinate of the hand centroid.

3. And obtaining the world coordinate of the centroid of the hand according to the mapping relation between the image coordinate and the world coordinate, wherein the coordinate is the hand position characteristic.

in order to extract the action characteristics, the hand characteristics in the preorder image need to be recorded. The hand motion characteristics in the period can be obtained by comparing the difference of one or more characteristics in the hand form, posture or position characteristics of the current frame and the preorder frame, and finally, the extraction result needs to be stored for subsequent image processing and analysis. As shown in fig. 10, the process of extracting the hand motion features in step two includes:

Step3, outputting a hand action characteristic result;

The hand shape, posture, position and motion characteristics can be separately used for gesture control, and can be combined according to actual needs.

when multiple features are adopted for control, the features can be fused into a comprehensive feature, for example, the following three features can be regarded as a combination, and when a gesture which simultaneously satisfies the following three features is detected by a user, a certain function of a certain device is directly executed:

(1) The palm is unfolded and stretched;

(2) the palm translates to the right front;

(3) the palm center turns from up to down.

in addition, each feature may also be associated with a different control level, for example, when the following features are detected, the corresponding instructions are executed respectively:

(1) When the palm is detected to be unfolded and stretched flatly, namely all the five fingers stretch out, selecting equipment with the number of 5;

(2) On the basis of the instruction 1, if the palm is further detected to translate forward, selecting a function 1 in the No. 5 device;

(3) on the basis of the instruction 2, if the palm is detected to be turned from up to down, executing a sub-function 1 in the No. 5 equipment function 1; if a palm roll is detected, sub-function 2 of device function No. 5 1 is executed.

In this way, the user can complete the device selection, the function selection and the function execution through one coherent gesture motion.

Example 2

reading image data of an input image;

Step four, storing the gesture recognition result of the step three;

the hand features in the second step are extracted by a feature extraction method of a depth image, as shown in fig. 11, the hand feature extraction process is as follows:

Step 2: segmenting a hand region input in the image using depth information;

Step 3: extracting hand feature points in the hand region;

Step 5: establishing a hand coordinate system;

Example 3

the embodiment provides a gesture control system for implementing the gesture control method according to claim 1, as shown in fig. 12, the technical solution adopted is as follows:

The main controller is used for controlling the operation of other modules and acquiring the operation state of each module in real time; the data acquisition module captures gesture actions in real time through a camera and transmits data to the data processing module; the data processing module is used for extracting hand features from the captured data, recognizing gesture commands and transmitting the gesture commands to the command output module; the command output module receives the gesture command and sends a function command to the corresponding equipment; the operation feedback module can provide feedback for a user in real time, so that the user can conveniently control the current system operation state.

according to the actual application requirements, the control system can adopt the following hardware implementation mode:

the main controller, data processing and instruction output module may be built based on an embedded platform or other computing platform with image processing capabilities. In order to ensure the real-time performance, safety and stability of the system operation, a hardware platform carrying the system needs to have the capabilities of multi-thread computing and parallel computing. Meanwhile, because the system needs to run continuously for a long time, reducing power consumption is an important factor of a hardware platform. Embedded platforms introduced by imperial viation corporation, such as TK1, TX1, and TX2 of Tegra series, can meet the hardware platform requirements of the system.

The data acquisition module may select a monocular or depth camera as the video image capture device. The monocular camera is a module consisting of a single sensor and a single lens, and the module has the function of acquiring a color image or a gray image of a scene in real time according to external driving. The depth camera mainly includes a sensor based on TOF (Time of flight), structured light or multi-angle imaging, and is mainly different from a monocular camera in that the depth camera can also capture the spatial distance between an object in a scene and the camera in real Time. The data acquisition module can be connected with the main controller and the data processing module through data transmission modes such as a USB data line, optical fibers or wireless and the like, and transmits data in real time.

the command output module is used as an interface between the gesture control system and the controlled device, and transmits the control command obtained by the analysis of the data processing module to the controlled device according to a specified communication protocol. The module can send instructions to the controlled equipment through a USB data line, an optical fiber or a short-distance wireless transmission protocol, wherein the short-distance wireless transmission protocol comprises ZigBee, NB-IOT, LoRA and the like. The protocol and the hardware module supporting the protocol are both components of the instruction output module.

The operation feedback module can provide feedback to the user through vibration, light or sound, and the like, and can comprise a vibration device, an LED, a loudspeaker or other hardware equipment with the same function according to actual requirements.

the gesture control method provided by the invention can realize gesture action recognition based on the monocular two-dimensional image or the depth image, and is wide in application range. The control system can adopt the monocular camera as the gesture motion capturing device, and the hardware implementation mode has the advantages of low cost, simple equipment and easy construction and maintenance. The invention can realize the selection and control of a plurality of groups of equipment and a plurality of functional modules in the equipment through a single coherent gesture, and can control a more complex hardware system. The gesture control method provided by the invention can realize blind operation, avoids the sight line of a user from shifting, and is beneficial to reducing the possibility of accidents compared with the traditional method when being used in application scenes such as automobile driving and the like.

Example 4

Embodiment 4 provides an automobile central control system based on the gesture control method of the present invention. The hardware device installation is shown in fig. 13. Wherein:

1. the gesture operation area is located at the center console and close to the gear lever.

2. the central control system adopts an embedded computing platform, the platform is integrated in the vehicle, and each device is controlled through a bus in the vehicle.

3. a monocular grayscale camera or a depth camera is used as a capturing device, and the camera is arranged between a steering wheel and a center console and comprises an infrared light supplement lamp to facilitate imaging in a low-illumination scene.

4. the vibration feedback device is arranged on the right side of the backrest of the driver seat. In addition, the system may also provide audio feedback to the user through the in-vehicle audio system.

The following describes the control method of the present invention, taking a driver's side window and a sunroof as examples.

the control method of the window control in the automobile central control system comprises the following steps: the thumb and index finger of the right hand are extended, and the thumb points upwards, as shown in fig. 14(a), the gesture is selected by the driver side window, the shape of the hand is kept unchanged, the hand is rotated to enable the palm to be downward, so that the window can be controlled to be lowered, and the palm to be upward, so that the window can be controlled to be raised. The method comprises the following specific steps:

Step a: stretching out a thumb and a forefinger of a right hand, pointing the thumb upwards, making a gesture for selecting the vehicle window of the driver side, selecting the vehicle window of the driver side as controlled equipment by the system, and sending vibration and sound feedback;

Step b: when the window is not in the fully-lowered state, the hand is rotated to the palm-down position, as shown in fig. 14(c), the system gives a window lowering instruction and sends vibration and sound feedback to the user, and the window on the driver side starts to descend, at this moment:

a) keeping the current hand shape and moving out of the operation area, or changing the hand shape, for example, restoring the hand to a relaxed state, the system will continue to execute the window descending instruction until the window is completely descended;

b) If the control gesture of the other device is made, the gesture control command of the other device is executed while the vehicle window descending instruction is continuously executed, and the system sends corresponding vibration and sound feedback;

c) Before the car window is completely descended, the hand is restored to the state shown in fig. 14(a), the system sends a car window static instruction and sends vibration and sound feedback to a user, and the car window is kept at the current position and stops descending;

step c: when the window is not in the completely lifted state, making a driver-side window selection gesture and rotating the hand to the palm-up direction, as shown in fig. 15(c), the system sends a window lifting instruction and sends vibration and sound feedback to the user, and the driver-side window starts to lift, at this time:

a) keeping the current hand shape and moving out of the operation area, or changing the hand shape, for example, restoring the hand to a relaxed state, the system will continue to execute the window ascending command until the window is completely lifted;

b) if the control gesture of the other device is made, the gesture control command of the other device is executed while the vehicle window ascending instruction is continuously executed, and the system sends corresponding vibration and sound feedback;

c) Before the window is completely lifted, the hand is restored to the state of fig. 15(a), the system sends out a window stop command, and sends vibration and sound feedback to a user, and the window keeps at the current position and stops lifting.

in the above process, step a can be skipped and step b or c can be directly executed, and the system will directly issue the corresponding command and feedback.

in a window control module in a central control system of an automobile, the opening and closing of an automobile skylight comprises two modes of translation and tilting, wherein the translation mode refers to that the skylight moves backwards to open and moves forwards to close, as shown in fig. 16 (a); the tilt mode refers to the skylight rear being raised open and lowered closed, as shown in fig. 16 (b).

The control mode is as follows: the thumb, index finger and middle finger of the right hand are extended, and the thumb points upward, as shown in fig. 17(a), and this gesture is a selected gesture for the skylight. Keeping the shape of the hand unchanged, rotating the hand to enable the palm to move downwards to control the skylight to be opened in a translation mode, and controlling the palm to move upwards to close; the left hand part is swung to control the skylight to be obliquely opened and the right hand part is controlled to be closed. The method comprises the following specific steps:

a translation mode:

step a: stretching out a thumb, an index finger and a middle finger of a right hand, pointing the thumb upwards, making a skylight selection gesture, selecting the skylight as controlled equipment by the system, and sending vibration and sound feedback;

Step b: when the skylight is in the translation mode and is not in the fully opened state, the hand is rotated to the palm downwards, as shown in fig. 17(c), the system sends out a skylight translation opening instruction and sends vibration and sound feedback to the user, and the skylight starts to translate backwards, at this time:

a) keeping the current hand shape and moving out of the operation area, or changing the hand shape, for example, restoring the hand to a relaxed state, the system will continue to execute the skylight translation opening instruction until the skylight is completely opened;

b) if a skylight tilt mode gesture is made, firstly, the skylight is translated and closed, then a tilt mode command is executed, and the system sends corresponding vibration and sound feedback;

c) If a control gesture of another device is made, while a skylight translation opening instruction is continuously executed, a gesture control command of the other device is executed, and a system sends corresponding vibration and sound feedback;

d) before the skylight is completely opened, the hand is restored to the state shown in the figure 17(a), the system sends a skylight static instruction and sends vibration and sound feedback to a user, the skylight is kept at the current position, and the backward movement is stopped;

step c: when the skylight is in the translation mode and is not in the completely closed state, making a skylight selection gesture and rotating the hand to the palm upward, as shown in fig. 18(c), the system sends a skylight translation closing instruction and sends vibration and sound feedback to the user, and the skylight starts to move forward, at this time:

a) Keeping the current hand shape and moving out of the operation area, or changing the hand shape, for example, restoring the hand to a relaxed state, the system will continue to execute the skylight translation closing instruction until the skylight is completely closed;

c) If the control gesture of the other device is made, the gesture control command of the other device is executed while the sliding and closing instruction of the skylight is continuously executed, and the system sends corresponding vibration and sound feedback;

d) Before the skylight is completely closed, the hand is restored to the state shown in fig. 18(a), the system sends a skylight static instruction and sends vibration and sound feedback to a user, and the skylight is kept at the current position and stops moving forwards.

the tilt mode:

step b: when the sun roof is in the tilt mode and not in the fully open state, the hand is swung to the left, as shown in fig. 19(c), the system issues a sun roof tilt open command and sends vibration and sound feedback to the user, and the rear part of the sun roof starts to lift up, at this time:

a) Keeping the current hand shape and moving out of the operation area, or changing the hand shape, for example, restoring the hand to a relaxed state, the system will continue to execute the skylight tilt-open command until the skylight is completely opened;

b) if a skylight translation mode gesture is made, firstly closing the skylight in an inclined mode, then executing a translation mode command, and sending corresponding vibration and sound feedback by the system;

c) If a control gesture of another device is made, a gesture control command of the other device is executed while a skylight tilt opening instruction is continuously executed, and a system sends corresponding vibration and sound feedback;

d) before the skylight is completely opened, the hand is restored to the state shown in the figure 19(a), the system sends a skylight static instruction and sends vibration and sound feedback to a user, and the skylight is kept at the current position and stops being lifted;

step c: when the sun roof is in the tilt mode and is not in the fully closed state, making a sun roof selection gesture and swinging the hand to the right, as shown in fig. 20(c), the system sends a sun roof tilt closing instruction and sends vibration and sound feedback to the user, and the rear part of the sun roof starts to descend, at this time:

a) keeping the current hand shape and moving out of the operation area, or changing the hand shape, for example, restoring the hand to a relaxed state, the system will continue to execute the skylight tilt closing instruction until the skylight is completely closed;

c) if a control gesture of another device is made, a gesture control command of the other device is executed while the skylight inclined closing instruction is continuously executed, and the system sends corresponding vibration and sound feedback;

d) before the skylight is completely closed, the hand is restored to the state shown in fig. 20(a), the system sends a skylight static instruction and sends vibration and sound feedback to a user, and the skylight is kept at the current position and stops descending.

example 5

embodiment 5 is an intelligent home central control device based on the gesture control method of the present invention, wherein a hardware device installation manner is shown in fig. 21. Wherein:

1. the gesture operation area is positioned at the right hand of the seat of the user.

2. The central control system adopts an embedded computing platform and is connected with the camera and each controlled device through wireless signals.

3. A monocular grayscale camera or a depth camera is used as a capturing device, and the camera is arranged above a television and comprises an infrared light supplement lamp to facilitate imaging in a low-illumination scene. According to the practical application scene, the camera can also be placed at other positions capable of clearly capturing the gesture actions of the user.

4. the system sends voice feedback to the user through the controlled speaker.

5. in the controlled equipment, two sound boxes can support the central control system to provide sound feedback, and are connected with a television to form multimedia equipment.

the control method of the present invention is described below by taking a floor lamp and a multimedia device as examples.

the control functions of the floor lamp comprise opening, closing and dimming. The control mode is as follows: the thumb and forefinger of the right hand are extended, and the thumb points upward, as shown in fig. 22(a), this gesture is a selected gesture of the floor lamp. The shape of the hand is kept unchanged, the floor lamp can be turned on by swinging the hand rightwards, and the floor lamp is turned off leftwards; the light can be controlled to become dark when the hand is rotated to make the palm face downward, and the light can be controlled to become bright when the hand is rotated to make the palm face upward.

(1) Floor lamp turn-on and turn-off control

Step a: stretching out the thumb and the index finger of the right hand, pointing the thumb upwards, making a gesture for selecting the floor lamp, selecting the floor lamp as a controlled device by the system, and sending sound feedback;

Step b: when the floor lamp is in the off state, the hand is swung to the right, as shown in fig. 22(c), the system sends out a floor lamp turn-on instruction, and sends voice feedback to the user, and the floor lamp is turned on;

step c: when the floor lamp is in the on state, the hand is swung leftward, as shown in fig. 23(c), the system sends a command to turn off the floor lamp, and sends a voice feedback to the user, so that the floor lamp is turned off.

(2) light and shade adjustment of floor lamp

step b: when the floor lamp is in the on state, the hand is rotated downwards towards the palm center, as shown in fig. 24(c), the system sends out a brightness reduction instruction of the floor lamp and sends a voice feedback to the user, the floor lamp begins to darken, at this moment:

a) The brightness change speed is determined by the rotation angle of the hand, the larger the rotation angle is, the faster the change speed is, and the system continuously reduces the brightness until the brightness is adjusted to the minimum;

b) when the hand shape is changed, the hand is restored to the state shown in fig. 24(a) or the hand is moved out of the operating area, the system sends out a command for stopping the brightness change of the floor lamp, and sends sound feedback to the user, and the floor lamp keeps the current brightness unchanged;

c) if a control gesture of another device is made, the system firstly sends a brightness change stopping instruction of the floor lamp, then executes a gesture control command of the other device, the floor lamp keeps the current brightness unchanged, and the system sends corresponding sound feedback;

Step c: when the floor lamp is in the on state, the hand is rotated upward toward the palm center, as shown in fig. 25(c), the system sends out a brightness increase instruction of the floor lamp, and sends a voice feedback to the user, and the floor lamp starts to turn on, at this moment:

a) The brightness change speed is determined by the rotation angle of the hand, the larger the rotation angle is, the faster the change speed is, and the system continuously increases the brightness until the brightness is adjusted to the maximum;

b) When the hand shape is changed, the hand is restored to the state shown in fig. 25(a) or the hand is moved out of the operating area, the system sends out a command for stopping the brightness change of the floor lamp and sends sound feedback to the user, and the floor lamp keeps the current brightness unchanged;

c) if a control gesture of another device is made, the system firstly sends a brightness change stopping instruction of the floor lamp, then executes a gesture control command of the other device, the floor lamp keeps the current brightness unchanged, and the system sends corresponding sound feedback.

Example 6

The embodiment provides a multimedia playing facility control device based on the gesture control method, and the functions of the multimedia device comprise opening, closing, playing, pausing, previous file, current file fast rewinding, next file, current file fast forwarding and volume adjustment. The multimedia device selection gesture is shown in fig. 26(a), and the user can extend the thumb, index finger and middle finger of the right hand, and the thumb points upward to select the multimedia device as the controlled device.

(1) multimedia device turning on and off

step a: stretching out the thumb, the index finger and the middle finger of the right hand, pointing the thumb upwards, making a multimedia device selection gesture, selecting the multimedia device as a controlled device by the system, and sending sound feedback;

Step b: when the multimedia device is in the closed state, the hand and the lower arm are lifted up as shown in fig. 26(c), and then the hand is rotated to the palm direction, as shown in fig. 26(f), the system sends a multimedia device opening instruction, and sends sound feedback to the user, and the multimedia device is opened.

Step c: when the multimedia device is in an open state, the hand and the forearm are lifted up as shown in fig. 27(c), and then the hand is rotated to the palm downward as shown in fig. 27(f), then the system sends a multimedia device closing instruction and sends sound feedback to the user, and the multimedia device is closed.

(2) multimedia equipment playing and pausing

Playing and pausing share the same gesture, and the control equipment is switched between two states. The method comprises the following specific steps:

step a: stretching out the thumb, index finger and middle finger of the right hand, pointing the thumb upwards, making a multimedia device selection gesture, as shown in fig. 28(a), selecting the multimedia device as a controlled device by the system, and sending sound feedback;

step b: when the device is in a pause state, the hand and the forearm are lifted up as shown in fig. 28(c), and then the original position is recovered as shown in fig. 28(f), the system sends a multimedia device playing instruction and sends sound feedback to the user, and the multimedia device starts playing;

step c: when the device is in a playing state, the hand and the forearm are lifted up as shown in fig. 28(c), and then the original position is recovered as shown in fig. 28(f), then the system sends a multimedia device pause instruction and sends sound feedback to the user, and the multimedia device pauses playing.

(3) multimedia equipment file switching and playing progress adjustment

step a: stretching out the thumb, index finger and middle finger of the right hand, pointing the thumb upwards, making a multimedia device selection gesture, as shown in fig. 29(a), selecting the multimedia device as a controlled device by the system, and sending sound feedback;

step b: when the media playing device is in an open state, the hand swings left and then returns to the original position, as shown in fig. 29, the system sends a command of switching the multimedia device to the previous file, and sends sound feedback to the user, and the multimedia device is switched to the previous file;

step c: when the media playing device is in an open state, the hand swings right and then returns to the original position, as shown in fig. 31, the system sends a command of switching the multimedia device to the next file, and sends sound feedback to the user, and the multimedia device is switched to the next file;

step d: when the media playing device is in an open state, the hand swings left and remains unchanged, as shown in fig. 30(c), the system sends a multimedia device current file fast-backward instruction, and sends a sound feedback to the user, and the multimedia device reverses the playing progress under the current playing file, at this time:

a) When the hand shape is changed, the hand is restored to the state of the graph 30(a) or the hand is moved out of the operation area, the system sends a multimedia device playing progress adjusting stop instruction and sends sound feedback to the user, and the multimedia device keeps the current playing progress;

b) if a control gesture of another device is made, the system firstly sends a playing progress adjusting and stopping instruction of the media playing device, then executes a gesture control command of the other device, the multimedia device keeps the current playing progress, and the system sends corresponding sound feedback;

Step e: when the media playing device is in the open state, the hand swings right and remains unchanged, as shown in fig. 32(c), the system sends a multimedia device current file fast forward instruction, and sends a sound feedback to the user, and the multimedia device moves forward the playing progress under the current playing file, at this time:

a) when the hand shape is changed, the hand is restored to the state of fig. 32(a) or the hand is moved out of the operating area, the system sends a media playing device playing progress adjusting stop instruction, sound feedback is sent to the user, and the multimedia device keeps the current playing progress;

b) If a control gesture of another device is made, the system firstly sends a playing progress adjusting and stopping instruction of the media playing device, then executes a gesture control command of the other device, the multimedia device keeps the current playing progress, and the system sends corresponding sound feedback.

in the above process, step a can be skipped, steps b to e are directly executed, and the system directly sends out corresponding instructions and feedback.

(4) Multimedia device volume adjustment

Step a: stretching out the thumb, index finger and middle finger of the right hand, pointing the thumb upwards, making a multimedia device selection gesture, as shown in fig. 33(a), selecting the multimedia device as a controlled device by the system, and sending sound feedback;

Step b: when the multimedia device is in the on state, the hand is rotated to the palm downward direction, as shown in fig. 33(c), the system sends out a multimedia device volume reduction instruction and sends a voice feedback to the user, the multimedia device volume starts to be reduced, at this time:

a) The volume change speed is determined by the rotation angle of the hand, the larger the rotation angle is, the faster the change speed is, and the system continuously reduces the volume until the volume is adjusted to the minimum;

b) when the hand shape is changed, the hand is restored to the state shown in fig. 33(a) or the hand is moved out of the operating area, the system sends a multimedia device volume adjustment stop instruction and sends sound feedback to the user, and the multimedia device keeps the current volume unchanged;

c) If a control gesture of another device is made, the system firstly sends a multimedia device volume adjustment stopping instruction, then executes a gesture control command of the other device, the multimedia device keeps the current volume unchanged, and the system sends corresponding sound feedback;

Step c: when the multimedia device is in the on state, the hand is rotated upward toward the palm center, as shown in fig. 34(c), the system sends out a multimedia device volume increase instruction and sends a voice feedback to the user, and the multimedia device volume starts to increase, at this time:

a) the volume change speed is determined by the rotation angle of the hand, the larger the rotation angle is, the faster the change speed is, and the system continuously increases the volume until the volume is adjusted to the maximum;

b) when the hand shape is changed, the hand is restored to the state shown in fig. 34(a) or the hand is moved out of the operating area, the system sends a multimedia device volume adjustment stop instruction, sound feedback is sent to the user, and the multimedia device keeps the current volume unchanged;

if a control gesture of another device is made, the system firstly sends a multimedia device volume adjustment stopping instruction, then executes a gesture control command of the other device, the multimedia device keeps the current volume unchanged, and the system sends corresponding sound feedback.

Although the present invention has been described with reference to the preferred embodiments, it should be understood that various changes and modifications can be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A gesture control method based on hand form, posture, position and motion characteristics is characterized in that a monocular camera is adopted as a gesture motion capture device, and gesture motion recognition is realized based on a monocular two-dimensional image or a depth image; the monocular camera is a module consisting of a single sensor and a single lens, and the module has the function of acquiring a color image or a gray image of a scene in real time according to external driving; the method comprises the following steps:

Reading image data of an input image;

Step four, storing the gesture recognition result of the step three;

step five, judging whether the gesture recognition result in the step four is a complete gesture, and if the gesture recognition result is the complete gesture, sending a gesture command corresponding to the gesture recognition result; if the gesture is not a complete gesture, iteratively executing the first step to the fourth step until the complete gesture is recognized;

extracting the hand morphological characteristics in the second step by an image fractal method or a template matching method or a method combining the image fractal method and the template matching method; when the hand morphological features are extracted by an image fractal method in the second step, the extraction process of the image fractal method comprises the following steps:

step1, reading image data of an input image;

step5, outputting the hand form feature result in the step 4;

And extracting the hand position features in the step two by using a hand morphological feature result, wherein the extraction process of the hand position features comprises the following steps:

Step A, reading the hand form feature result;

C, outputting the hand position feature result in the step B;

1. in the process of establishing a world coordinate system, the mapping relation between the image coordinates of the image captured by the camera and the world coordinates is included;

2. after the hand edge contour is obtained through extraction, calculating the average value of the image coordinates of each point on the edge contour, namely the image coordinates of the hand centroid;

2. The gesture control method according to claim 1, wherein the hand feature is described in a world coordinate system and a hand coordinate system; the world coordinate system o-xyz is taken as an absolute coordinate system of the whole system; when the hand coordinate system o ' -x ' y ' z ' is unfolded by stretching the palm, the intersection point of the straight line of the thumb and the straight line of the middle finger is the original point, the straight line of the middle finger is the x ' axis, and the direction pointed by the finger tip is the positive direction; the direction vertical to the palm is the y' axis direction, when the right hand is used as the control hand, the palm direction is the forward direction, and when the left hand is used, the back direction is the forward direction; the z ' axis is perpendicular to the plane formed by the x ' and y ' axes, and the thumb direction is the positive direction.

3. the gesture control method according to claim 1, wherein the hand morphology features of step two are extracted by a template matching method, and the extraction process of the template matching method comprises:

4. The gesture control method according to claim 1, wherein the hand gesture feature extraction process in step two comprises:

And d, outputting a characteristic result of the hand posture characteristic.

5. the gesture control method according to claim 1, wherein the hand position features of step two are extracted by using hand posture feature results, and the hand position feature extraction process comprises:

step I, reading the hand gesture feature result

and III, outputting the hand position feature result in the step II.

6. the gesture control method according to claim 1, wherein the extracting process of the hand motion features in the second step comprises:

step3, outputting a hand action characteristic result;

7. The gesture control method according to claim 1, wherein the hand features of step two are extracted by a depth image feature extraction method, and the extraction process is as follows:

step 2: segmenting a hand region input in the image using depth information;

step 3: extracting hand feature points in the hand region;

step 5: establishing a hand coordinate system;

8. A gesture control system for implementing the gesture control method according to claim 1, wherein the control system comprises a main controller, a data acquisition module, a data processing module, an instruction output module and an operation feedback module; the data acquisition signal control interaction end of the main controller is connected with the data control interaction end of the data acquisition module; the data processing signal control interaction end of the main controller is connected with the data control interaction end of the data processing module; the command output signal control interaction end of the main controller is connected with the data control interaction end of the command output module; an operation feedback signal control interaction end of the main controller is connected with a data control interaction end of the operation feedback module; the data output end of the data acquisition module is connected with the data input end of the data processing module; the data output end of the data processing module is connected with the data input end of the instruction output module; the control system adopts a monocular camera or a depth camera as a gesture motion capturing device, and realizes gesture motion recognition based on a monocular two-dimensional image or a depth image.