CN104700403B - A kind of gesture based on kinect controls the Virtual Demonstration method of hydraulic support - Google Patents

Abstract

The invention discloses a kind of gesture based on kinect and control virtual teaching system and the method for hydraulic support, including kinect module, depth of field data acquisition module, articulare message processing module, gesture feature matching module, control module and display apparatus module, kinect module is connected with depth of field data acquisition module, articulare message processing module, gesture feature matching module, control module and display apparatus module successively.Operator are only needed to make certain gestures, it is possible to observe virtual hydraulic support by display apparatus module and make corresponding action, it is simple to people are better understood from the work process of hydraulic support.

Description

A kind of gesture based on kinect controls the Virtual Demonstration method of hydraulic support

Technical field

The present invention relates to a kind of gesture based on kinect and control virtual teaching system and the method for hydraulic support, belong to automatic control technology field.

Background technology

Human-computer interaction technology refers to the input and output device by computer, it is achieved people exchanges with computer.Common man-machine interaction mode is the system using the position to staff and pose to carry out real-time tracking, this kind equipment has electromagnetic tracking device, inertial sensor and data glove etc., these belong to the sensor of contact, need to dress extra equipment and make interaction the most natural.And the body feeling interaction equipment kinect of Microsoft is contactless sensor, computer can be controlled by the hand motion of operator, it is not necessary to dress extra equipment on the person so that people can interact with a computer in a natural manner.

Hydraulic support is used to control the works of coal-face mine pressure, and as the necessary equipment of comprehensive mechanical coal mining, it is coal mining machine in full-mechanized mining face, drag conveyor and staff provide safe work space.But people are difficult to the work process of the study understanding hydraulic support of image.In order to improve people's interactive experience sense to hydraulic support work process, need to design the intersection control routine of a kind of people-hydraulic support virtual three-dimensional environment, make people be better understood from the work process of hydraulic support.

Summary of the invention

The problem existed for above-mentioned prior art, the present invention provides virtual teaching system and the method for a kind of gesture based on kinect control hydraulic support, operator are only needed to make certain gestures, just can observe virtual hydraulic support by display apparatus module and make corresponding action, it is simple to people are better understood from the work process of hydraulic support.

To achieve these goals, the technical solution used in the present invention is: this kind gesture based on kinect controls the virtual teaching system of hydraulic support, including kinect module, depth of field data acquisition module, articulare message processing module, gesture feature matching module, control module and display apparatus module, kinect module is connected with depth of field data acquisition module, articulare message processing module, gesture feature matching module, control module and display apparatus module successively.

This kind gesture based on kinect controls the Virtual Demonstration method of hydraulic support, and it comprises the concrete steps that:

(1) preset eight different standard gestures, and eight gestures are stored in control module；

(2) control module is by the fall post of eight corresponding hydraulic supports of standard gestures difference, shifting frame, setting prop, pushing and sliding, fall face guard, liter face guard, receipts front-axle beam and the control signal rising eight actions of front-axle beam；

(3) initializing kinect module, set the video camera elevation angle, standing stands between distance 1.2 meters to 3.5 meters scopes of photographic head, one of eight standard gestures making setting；

(4) action of kinect module detection operator, and from color stream, deep stream and skeleton stream, obtain skeleton information data frame；

(5) the skeleton information data frame that depth of field data acquisition module analyzing and processing kinect module transmits, obtain image depth data, extract skeleton information, by setting up the 3D coordinate system of human hand and arm joint, obtain human hand and arm joint node coordinate, the position different to identify human body；

(6) articulare message processing module is by the 3D coordinate of obtained human hand and arm joint, calculates the rotational angle on human body shoulder joint three degree of freedom direction, the rotational angle on two degree of freedom directions of elbow joint；According to human body shoulder joint and elbow joint rotational angle information, identify the rotation information of human arm skeleton node, carry out data process by catching the change of different skeletal joint point angle；

(7) each above-mentioned different joint angles information is carried out characteristic matching with each joint angles information of eight standard gestures of storage in control module by gesture feature matching module；If mating unsuccessful, then returning step (3), if the match is successful, then gesture feature matching module passes data to control module；

(8) control module receives gesture feature matching module and transmits the data that the match is successful, after analyzing and processing, send the action control signal of the hydraulic support corresponding with this gesture, and then control three-dimensional hydraulic support action, and shown by display apparatus module.

Compared with prior art, the present invention carries out the collection of gesture information by kinect module, utilization can extract the position in the three-dimensional information tracking each joint of arm of skeleton point, by recognizing specific gesture, control virtual hydraulic support and make corresponding action, it is simple to people are better understood from the work process of hydraulic support.

Accompanying drawing explanation

Fig. 1 is the electric theory diagram of the present invention；

Human arm motion's schematic diagram when Fig. 2 is to make gesture in the present invention；

Fig. 3 is the D-H coordinate system of arm in the present invention；

Fig. 4 is the flow chart of the present invention.

Detailed description of the invention

The invention will be further described below in conjunction with the accompanying drawings.

As depicted in figs. 1 and 2, this kind gesture based on kinect controls the virtual teaching system of hydraulic support, including kinect module, depth of field data acquisition module, articulare message processing module, gesture feature matching module, control module and display apparatus module, kinect module is connected with depth of field data acquisition module, articulare message processing module, gesture feature matching module, control module and display apparatus module successively.

This kind gesture based on kinect controls the Virtual Demonstration method of hydraulic support, and it comprises the concrete steps that:

(1) preset eight different standard gestures, and eight gestures are stored in control module；

(2) control module is by the fall post of eight corresponding hydraulic supports of standard gestures difference, shifting frame, setting prop, pushing and sliding, fall face guard, liter face guard, receipts front-axle beam and the control signal rising eight actions of front-axle beam；

(3) initializing kinect module, set the video camera elevation angle, standing stands between distance 1.2 meters to 3.5 meters scopes of photographic head, one of eight standard gestures making setting；

(4) action of kinect module detection operator, and from color stream, deep stream and skeleton stream, obtain skeleton information data frame；

(5) the skeleton information data frame that depth of field data acquisition module analyzing and processing kinect module transmits, obtain image depth data, extract skeleton information, by setting up the 3D coordinate system of human hand and arm joint, obtain human hand and arm joint node coordinate, the position different to identify human body；

(6) articulare message processing module is by the 3D coordinate of obtained human hand and arm joint, calculates the rotational angle on human body shoulder joint three degree of freedom direction, the rotational angle on two degree of freedom directions of elbow joint；According to human body shoulder joint and elbow joint rotational angle information, identifying the rotation information of human arm skeleton node, carry out data process by catching the change of different skeletal joint point angle, concrete process is:

As it is shown on figure 3, the motion of human body right arm shoulder joint to be divided into flexion and extension, telescopic motion, rotary motion, respectively corresponding angle α₁、β₁、γ₁.Human elbow motion is flexion and extension, and corresponding angle is δ₁.Frame of reference o is set up at the intermediate point of human body both shoulders joint₀-x₀y₀z₀If the 3D coordinate of human body right arm shoulder joint skeleton node is (x₁,y₁,z₁), the 3D coordinate of right arm elbow joint skeleton node is (x₂,y₂,z₂), the 3D coordinate of right arm carpal joint skeleton node is (x₃,y₃,z₃)；

Right arm shoulder breadth $L_0=\sqrt{x_1^2+y_1^2+z_1^2}$

The big brachium of right arm $L_1=\sqrt{{(x_2-x_1)}^2+{(y_2-y_1)}^2+{(z_2-z_1)}^2}$

The little brachium of right arm $L_2=\sqrt{{(x_3-x_2)}^2+{(y_3-y_2)}^2+{(z_3-z_2)}^2}$

For shoulder joint flexor ,-40 ° of ＜ α₁＜ 90 °,

${\mathrm{\α}}_1=\arctan \left(\frac{y_2-y_1}{x_2-x_1}\right)$

For shoulder exhibition receipts action ,-90 ° of ＜ β₁＜ 20 °,

${\mathrm{\β}}_1=\arctan \left(\frac{z_2-z_1}{x_2-x_1}\right)$

For elbow joint flexor, 0 ° of ＜ δ₁＜ 130 °,

${\mathrm{\δ}}_1=\arccos \frac{[(x_1-x_2)\×(x_2-x_3)+(y_1-y_2)\×(y_2-y_3)+(z_1-z_2)\×(z_3-z_2)]}{L_1\×L_2}$

For shoulder joint rotary motion, can not directly be obtained by known coordinate.Here with setting up human synovial D-H coordinate system, draw the coordinate expressions of carpal joint node according to principle of coordinate transformation, then release γ according to known carpal joint node coordinate is counter₁。

According to the D-H parameter in table 1, write out the affine transformation matrix that each joint is corresponding.

$A_1=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& L_0\\ 0& 0& 0& 1\end{array}\right]$

$A_2=\left[\begin{array}{cccc}\cos {\mathrm{\θ}}_2& 0& \sin {\mathrm{\θ}}_2& 0\\ \sin {\mathrm{\θ}}_2& 0& -\cos {\mathrm{\θ}}_2& 0\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$

$A_3=\left[\begin{array}{cccc}\cos {\mathrm{\θ}}_3& 0& \sin {\mathrm{\θ}}_3& 0\\ \sin {\mathrm{\θ}}_3& 0& -\cos {\mathrm{\θ}}_3& 0\\ 0& 1& 0& 0\\ 0& 0& 0& 1\end{array}\right]$

$A_4=\left[\begin{array}{cccc}\cos {\mathrm{\θ}}_4& 0& \sin {\mathrm{\θ}}_4& 0\\ \sin {\mathrm{\θ}}_4& 0& -\cos {\mathrm{\θ}}_4& 0\\ 0& 1& 0& L_1\\ 0& 0& 0& 1\end{array}\right]$

$A_5=\left[\begin{array}{cccc}\cos {\mathrm{\θ}}_5& -\sin {\mathrm{\θ}}_5& 0& L_2\cos {\mathrm{\θ}}_5\\ \sin {\mathrm{\θ}}_5& \cos {\mathrm{\θ}}_5& 0& L_2\sin {\mathrm{\θ}}_5\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$

Obtaining total transformation matrix is

$T=A_1{A}_2{A}_3{A}_4{A}_5=\left[\begin{array}{cccc}{n}_x& o_x& a_x& p_x\\ n_y& o_y& a_y& p_y\\ n_z& o_z& a_z& p_z\\ 0& 0& 0& 1\end{array}\right]$

Wherein

x₃=p_x=L₁cosθ₂sinθ₃+L₂cosθ₅(sinθ₂sinθ₄+cosθ₂cosθ₃cosθ₄)+L₂cosθ₂sinθ₃sinθ₅

y₃=p_x=L₁sinθ₂sinθ₃-L₂cosθ₅(cosθ₂sinθ₄+cosθ₃cosθ₄sinθ₂)+L₂sinθ₂sinθ₃sinθ₅

z₃=L₀-L₁cosθ₃-L₂cosθ₃sinθ₅+L₂cosθ₄cosθ₅sinθ₃

Wherein

$\left\{\begin{array}{c}{\mathrm{\θ}}_2={\mathrm{\α}}_1-\mathrm{\π}/2\\ {\mathrm{\θ}}_3={\mathrm{\β}}_1-\mathrm{\π}/2\\ {\mathrm{\θ}}_4={\mathrm{\γ}}_1-\mathrm{\π}/2\\ {\mathrm{\θ}}_5=\mathrm{\δ}1\end{array}\right.$

According to known angle and length, substitute in above-mentioned formula, the rotary motion angle γ of shoulder joint can be obtained₁(-90°<γ₁<45°)；

(7) each above-mentioned different joint angles information is carried out characteristic matching with each joint angles information of eight standard gestures of storage in control module by gesture feature matching module；If mating unsuccessful, then returning step (3), if the match is successful and continues 5 seconds, then gesture feature matching module passes data to control module；

(8) control module receives gesture feature matching module and transmits the data that the match is successful, after analyzing and processing, send the action control signal of the hydraulic support corresponding with this gesture, and then control three-dimensional hydraulic support action, and shown by display apparatus module；Table 2 is the hydraulic pressure direct action that each different gesture feature is corresponding.

Table 1:

i	α	a	d	θ
					1	0	0	L0	0
2	90	0	0	θ2
					3	90	0	0	θ3
4	90	0	L1	θ4
					5	0	L2	0	θ5

Table 2:

	α1	β1	γ1	δ1	Action
						1	70	-70	0	90	Fall post
2	30	0	-90	90	Move frame
						3	0	-90	0	90	Setting prop
4	0	-90	-90	90	Pushing and sliding
						5	0	-90	45	0	Fall face guard
6	90	0	45	90	Rise face guard
						7	90	0	45	0	Receive front-axle beam
8	90	-90	0	0	Rise front-axle beam

Claims (1)

1. the Virtual Demonstration method of gesture based on a kinect control hydraulic support, it is characterised in that its concrete steps It is:

(1) preset eight different standard gestures, and eight gestures are stored in control module；

(2) control module is by the fall post of the most corresponding for eight standard gestures hydraulic support, shifting frame, setting prop, pushing and sliding, fall wall supporting Plate, liter face guard, receipts front-axle beam and the control signal of liter eight actions of front-axle beam；

(3) initializing kinect module, set the video camera elevation angle, standing stands in distance photographic head 1.2 meters to 3.5 Between rice scope, one of eight standard gestures making setting；

(4) action of kinect module detection operator, and from color stream, deep stream and skeleton stream, obtain skeleton Information data frame；

(5) the skeleton information data frame that depth of field data acquisition module analyzing and processing kinect module transmits, it is thus achieved that image Depth of field data, extracts skeleton information, by setting up the 3D coordinate system of human hand and arm joint, obtains human hand and arm joint Node coordinate, the position different to identify human body；

(6) articulare message processing module is by the 3D coordinate of obtained human hand and arm joint, calculates human body shoulder joint Rotational angle on joint three degree of freedom direction, the rotational angle on two degree of freedom directions of elbow joint；According to human body shoulder joint With elbow joint rotational angle information, identify the rotation information of human arm skeleton node, by catching different skeletal joint point The change of angle carries out data process；

(7) gesture feature matching module is by each above-mentioned different joint angles information and eight marks of storage in control module Each joint angles information of quasi-gesture carries out characteristic matching；If mating unsuccessful, then return step (3), if the match is successful, Then gesture feature matching module passes data to control module；

(8) control module receives gesture feature matching module and transmits the data that the match is successful, after analyzing and processing, send with The action control signal of the hydraulic support that this gesture is corresponding, and then control three-dimensional hydraulic support action, and by display Device module shows.