CN115754956A - Millimeter wave radar gesture recognition method based on envelope data time sequence - Google Patents
Millimeter wave radar gesture recognition method based on envelope data time sequence Download PDFInfo
- Publication number
- CN115754956A CN115754956A CN202211312638.XA CN202211312638A CN115754956A CN 115754956 A CN115754956 A CN 115754956A CN 202211312638 A CN202211312638 A CN 202211312638A CN 115754956 A CN115754956 A CN 115754956A
- Authority
- CN
- China
- Prior art keywords
- data
- gesture
- radar
- millimeter wave
- envelope
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Radar Systems Or Details Thereof (AREA)
Abstract
A millimeter wave radar gesture recognition method based on an envelope data time sequence belongs to the technical field of millimeter wave radars. The method comprises the following steps: step S1: acquiring data of different gestures by using a millimeter wave radar to form envelope data, wherein the envelope data of each gesture is two-dimensional matrix data in a frame sequence-distance unit organization form; step S2: preprocessing the envelope data read in the step S1; and step S3: putting the data preprocessed in the step S2 into an LSTM network for feature extraction; and step S4: and obtaining a gesture recognition result through a classifier of a full connection layer. The invention provides a gesture recognition method for dividing envelope data into time sequences based on LSTM, which collects gesture data by using single-transmitting single-receiving extremely narrow pulse radar and carries out gesture recognition classification by using the method, thereby realizing higher gesture recognition accuracy by using simplest data, more reliable algorithm and lower calculation complexity.
Description
Technical Field
The invention belongs to the technical field of millimeter wave radars, and particularly relates to a millimeter wave radar gesture recognition and classification problem based on an envelope data time sequence.
Background
With the development of radar technology and the cost reduction of radar systems, the application of radars in daily life is more and more extensive. Taking heartbeat detection as an example, a professional heartbeat detection system (electrocardiograph) is often complex and needs to be operated by medical staff; heartbeat detection systems (blood pressure meters, intelligent watches, and the like) used by ordinary people in daily life are generally expensive. The heartbeat detection system based on the millimeter wave radar is very low in cost and has high reliability. The palm-sized radar can be detected only by placing the palm-sized radar at a specific position, and many other functions such as breath detection and even gesture recognition can be added. And the system based on the millimeter wave radar is not influenced by ambient illumination, has extremely low power consumption and strong reliability, and can work all the day and night. In addition, high frequency radar signals have high penetration and have incomparable advantages compared with systems based on low frequency signals. The millimeter wave radar has the characteristics of small volume and easiness in integration, can be easily embedded into other equipment, and greatly expands the use scene. For a macro scene such as target detection, the radar detection range is wide, and the radar is not easily interfered by shielding objects; for a microscopic scene such as gesture recognition, the radar recognition precision is high, and the anti-jamming capability is strong. The millimeter wave radar system can be integrated in most mobile equipment (mobile phones) and can also be made into external expansion equipment, and the application scene is very flexible.
In many applications of millimeter wave radar, gesture recognition is relatively complex to implement. The current gesture recognition algorithms are mainly divided into two-dimensional and three-dimensional. The two-dimensional gesture recognition algorithm mainly relates to an image processing technology, and realizes gesture classification by using a computer vision-based method; and three-dimensional gesture recognition has still more depth information than two-dimensional data, and millimeter wave radar can obtain the three-dimensional echo data of gesture very easily. Therefore, in recent years, various gesture recognition methods based on millimeter wave radar have been widely studied. Most used among these are gesture recognition algorithms based on deep learning neural networks.
Traditional gesture recognition algorithms based on neural networks generally use multi-transmit multi-receive multi-channel radar systems. In general, a Linear Frequency Modulation (LFM) radar with high accuracy is used for transmitting and receiving, and signal processing, such as filtering, fast Fourier Transform (FFT), short Time Fourier Transform (STFT), and the like, is performed on received data; and then, on the basis that a large amount of data are used as training samples, a neural network is built to analyze and process the data, so that the task of gesture classification is realized. Among them, the use of Convolutional Neural Networks (CNN) can be said to be very widespread. The gesture recognition method is high in accuracy and good in robustness, but an LFM radar system is relatively expensive, a signal processing algorithm is relatively complex, and the method is not suitable under the condition that the computational power and the cost of an integrated system are limited; the training cost of the CNN-based deep learning model varies greatly according to the depth and complexity of the network, and the CNN itself has some problems when using envelope data:
1. the convolution is directly carried out on the two-dimensional data of the distance unit-frame sequence, so that the overfitting is easy under the condition of small data quantity, the accuracy of the judgment is limited, and meanwhile, the robustness of the model is also influenced.
2. The gesture envelope data is essentially sequence data which changes regularly in time, and the CNN often ignores the dependency of the data on the time dimension, further making the network less universal and less interpretable.
Disclosure of Invention
Aiming at the advantages and disadvantages of the traditional gesture recognition algorithm based on CNN, the invention provides a gesture recognition method for dividing envelope data into time sequences based on LSTM, single-shot single-receive ultra-narrow pulse radar is used for collecting gesture data and the method is used for gesture recognition and classification, and the highest gesture recognition accuracy is realized by using the simplest data, a more reliable algorithm and lower calculation complexity.
A millimeter wave radar gesture recognition method based on an envelope data time sequence comprises the following steps:
step S1: acquiring data of different gestures by using a millimeter wave radar to form envelope data, wherein the envelope data of each gesture is two-dimensional matrix data in a frame sequence-distance unit organization form;
step S2: preprocessing the envelope data read in the step S1;
and step S3: putting the data preprocessed in the step S2 into an LSTM network for feature extraction;
and step S4: and obtaining a gesture recognition result through a classifier of the full connection layer.
Preferably, the millimeter wave radar is connected with a PC through a USB interface, and the data is read and stored by using a radar data receiving upper computer program; the USB-A radar platform for serial port communication is used during gesture datase:Sub>A collection, and the USB-A platform for serial port communication and the USB-C platform based on the HID communication protocol are used during testing.
Preferably, the frame length f of each envelope datum is 10-30 frames, and the acquisition time of each gesture is 2-3 seconds under the data transmission rate of serial port communication; the number of distance units per data is D 0 (ii) a And after the data are collected by an upper computer program, the data are packaged and stored into a txt or mat format for subsequent processing.
Preferably, when gesture data are collected, a hand is placed in front of the millimeter wave radar, and the radar is placed on a table or supported on a palm; corresponding gesture actions are made within a certain time, pulse radar signals are received by a receiving antenna after being reflected by hands and are preprocessed in a radar chip, and then generated envelope data are directly transmitted back to a PC (personal computer) end and are packaged and stored through an upper computer program.
Preferably, the millimeter wave radar of the present invention is any one-shot, one-shot PCR radar or extremely narrow pulse radar.
Preferably, the preprocessing process of the present invention includes, but is not limited to, distance dimension down-sampling, normalization, and size unification, and the three data preprocessing methods are specifically implemented as follows:
(1) normalizing all data according to the maximum value of the envelope amplitude;
(2) downsampling all data to D in the distance dimension down ；
(3) Upsampling all data to F in frame order dimension up Namely, the data size normalization is realized by two steps (2) and (3).
Preferably, the LSTM network part for extracting features of the present invention freely adjusts the network parameters according to the size of the envelope data and the signal characteristics of the radar, and the parameters of the LSTM network part are defined as follows:
the characteristic number of the input data is H in ＝D down I.e. the length of the input data in the distance dimension; input time series length L in ＝F up Dividing the envelope data into time sequences according to a frame sequence and inputting the time sequences into a recurrent neural network; the number of hidden layer features is H cell The characteristic number of the matrix data output by the network is regarded as the distance dimension length of the output data, and the parameter is adjusted according to the actual performance of the network; the characteristic number of the output data is H out ＝H cell (ii) a RNN layer number D R (ii) a Group size N B (ii) a Using a unidirectional LSTM network; the data output by the network is a matrix with the size of (N) B ,L in ,H out ) (ii) a And other parameters are set and adjusted automatically according to needs.
Preferably, the invention is used for classified full-connection layer networkPartly without special form, using active layer, BN layer, dropout layer for improving network performance, classifier of full connection layer maps data output by LSTM network to one dimension tensor G 1×M Wherein M is the number of gesture types to be recognized; g 1×M Each value of the tensor corresponds to the probability g of the envelope data discriminating a gesture of a certain type i Namely, the judgment result is:
wherein i is the number of the identified gesture.
The invention only needs to use the simplest single-transmitting single-receiving PCR radar (pulse coherent radar) or extremely narrow pulse radar, which has low cost and sufficient reliability and identification precision for gesture identification. Although the pulse radar can also acquire radar data (envelope data, IQ data and the like) in different forms, the invention only uses the simplest envelope data, the data processing is simpler, and the contained information can meet the simple gesture classification.
And the present invention uses a long short term memory network (LSTM) based on a Recurrent Neural Network (RNN). Although the conventional RNN has a better performance on time series data, when data information to be used is too far away from the current node, the RNN loses the learning capability and thus the network effect is reduced. It is therefore desirable to use LSTM networks, a special form of RNN, which can learn long-term dependency information without significant learning cost and expense. The envelope data, although essentially two-dimensional image data, can be trained using an LSTM network if it is divided in time series.
The identification method of the invention is much simpler in technical principle. Firstly, the method has lower requirements on the radar system, only needs the simplest single-transmitting single-receiving system and has lower cost. Secondly, the method only needs simple preprocessing on the obtained gesture envelope data, and does not need a complex signal processing algorithm. Finally, the LSTM network based on the time sequence method is used for data training and gesture recognition classification, the overall calculation complexity is low, the accuracy rate is high, the robustness is good, the recognition speed is high, and the method can be applied to integrated systems of various levels.
Drawings
FIG. 1a is a schematic diagram of a gesture approaching from the front.
FIG. 1b is a schematic diagram of a gesture of moving away from the front.
FIG. 1c is a schematic diagram of a hand waving back and forth at a certain distance from the front.
FIG. 1d is a schematic diagram of a fist making gesture at a certain distance right in front.
Fig. 2 is a schematic structural diagram of an LSTM network used in the method.
Fig. 3 is a schematic diagram of a model structure based on an LSTM network used in the method.
Fig. 4 is an envelope map (two-dimensional, amplitude normalized) of one set of envelope data acquired by a radar.
Fig. 5 is an envelope map (three-dimensional, amplitude not normalized) of one set of envelope data acquired by a radar.
Detailed Description
The present invention will be further described with reference to the accompanying drawings.
The invention relates to a gesture recognition algorithm, which is based on a simple transmitting-receiving PCR radar system. The A111 radar of Acconeer is used, and has three preset data acquisition modes: envelope (Envelope), IQ, sparse (Sparse). The invention adopts an Envelope (Envelope) mode. The data that the radar was gathered are passed back PC in real time and are carried out data storage by the host computer procedure, and the radar chip only is used for receiving and dispatching signal data collection, and data processing all accomplishes at the PC end. All experimental procedures need not be completed in a special microwave dark room.
The present invention uses only envelope data, since the radar has limited data available. In combination with the performance and practical application of the radar, 4 simple gesture actions with high discrimination are designed in the example, and the actions comprise motion gestures with different directions and amplitudes. The basic gestures designed by the invention comprise: as shown in fig. 1 a-1 d, no radar is connected in any of fig. 1 a-1 d corresponding to a gesture of approaching from the front of the radar, a gesture of moving away from the front of the radar, a gesture of waving a hand back and forth at a certain distance from the front of the radar, and a gesture of making a fist at a certain distance from the front of the radar.
Since the millimeter wave radar parameters have been previously configured, and the detailed parameters of the radar have been given in the given description of use, a detailed description thereof will not be given here. The identification method of the invention does not need to use a plurality of parameters of the radar, and only needs to adjust the effective measurement range of the radar, the distance unit number in the range is effective, and the radar measurement range in the measurement of the method is about 10 cm-30 cm. The range bin of the PCR radar has been measured to be 0.5mm in previous tests. Setting a baud rate of 92160 in an upper computer program for collecting gesture data, setting the upper limit of a chirp signal of single-time packed data to be 30 (actually collected to be 29 frames), setting the total number of distance units to be 414, setting data storage formats to be txt and mat, and setting used data to be Envelope (Envelope) data. The integrity of gesture motion acquisition can be guaranteed through testing 30 frames of data for multiple times.
Pulse radar signals are received by a receiving antenna after being reflected by hands and are preprocessed in a radar chip, and then generated envelope data are directly transmitted back to a PC (personal computer) end and are packaged and stored through an upper computer program without other data processing. The collected gesture data is saved locally in txt and mat formats. The mat data is stored as matrix data of f 414 (f is more than 10 and less than 30), and can be directly opened by Matlab software; the txt file is a text document, with a single distance cell of data per frame in one line, and 414 data per frame starting with '$' and ending with '#'. Radar data of one-time gesture behaviors are packaged in the same file, namely, each file contains f frame data of the same gesture. And repeatedly acquiring to obtain a plurality of sample data of a plurality of gestures, storing the sample data under corresponding folder paths according to the gesture classification, and dividing the sample data into a training set and a test set, thereby facilitating subsequent processing.
Firstly, reading a gesture mat file by using a matlab, and drawing an envelope graph for analyzing data. The mat data is organized according to the format of each chirp sampling number (distance unit number) multiplied by chirp number to obtain a matrix S of a distance unit-frame sequence raw The matrix isIs a matrix of f 414. In the range dimension, the nth column of data characterizes the range radar n · Δ R + R 0 The larger the amplitude is, the stronger the reflected wave signal at the position is marked, namely the effective reflecting surface of the object is near the position; in the time sequence dimension, the mth row of data represents the envelope amplitude of the radar in the whole distance dimension of the m.DELTA.T slow time. Wherein Δ R is the distance unit, R 0 For range bin offset (i.e., radar start sweep distance), Δ T is the frame duration, i.e., slow time, of each frame sample of the PCR radar.
To S raw The original envelope data of (1) directly draws two-dimensional and three-dimensional graphs of the range unit-frame sequence, and the images drawn by using a certain waving data are shown in fig. 4 and 5. The two-dimensional envelope graph is an image with an envelope amplitude value normalized, a distance unit-normalized amplitude value graph is drawn for each frame, and a two-dimensional distance unit-frame sequence graph for converting the amplitude value into a color is drawn; the three-dimensional graph is not subjected to normalization processing, the x axis is a distance unit, the y axis is a frame sequence, and the z axis is an envelope data amplitude value. From the two-dimensional image of fig. 4, it can be known that the range bin of the hand-waving data slightly shakes with time, and from the three-dimensional image of fig. 5, it can be known that the echo energy of the hand-waving data greatly changes with time, which can be used as a basis for determining the hand-waving motion, and the rest of the gesture data are the same.
The gesture recognition method partially uses Python to program, the used machine learning library is Pythroch, the Pythroch library has a libtorch library of a C language version, and a complete Python-C + + interface is provided. The data used is mat data. The script program written by Python can be easily called by other programs and can also be converted into C language, thereby facilitating embedded development.
A millimeter wave radar gesture recognition method based on envelope data time sequence,
(1) The used millimeter wave radar is a finished product of an A111 radar chip of Acconeer, relevant parameters are all tested when the millimeter wave radar is used, and some parameters can be automatically modified in subsequent use through an official interface program. The radar chip is connected with the PC through a USB interface, and data can be read and stored by using an A111 development board radar data receiving upper computer program. In the method, ase:Sub>A USB-A radar platform for serial port communication is used during gesture datase:Sub>A acquisition, and ase:Sub>A USB-A platform for serial port communication and ase:Sub>A USB-C platform based on an HID communication protocol are used during testing.
(2) The radar used in the present invention does not require data acquisition in a special darkroom. When collecting gesture data, the tester places the hand directly in front of the millimeter wave radar, which may be placed flat on a table or held in the palm of the hand. The tester makes corresponding gesture motion within a certain time, and then generated envelope data are directly transmitted back to the PC end and are packaged and stored into a txt or mat format through an upper computer program for subsequent processing. Considering that the time of different gestures of different people is different, the frame length f of each data is between 10 and 30 frames, and the acquisition time of each gesture is 2 to 3 seconds under the data transmission rate of serial port communication; the number of distance units per data is D 0 . The envelope data of each gesture is two-dimensional matrix data in the form of frame order-distance unit organization.
(3) And classifying and storing all collected envelope data. All mat data files are read by Python, and the used machine learning library is a Pythrch library. And (4) building a neural network based on the LSTM for training, and using the finally trained network model for testing. Firstly, preprocessing the read envelope data, then putting the preprocessed data into an LSTM network for feature extraction, and finally obtaining a gesture recognition result through a classifier of a full connection layer (FC layer).
(4) Methods of data preprocessing include, but are not limited to, distance dimension down-sampling, normalization, size normalization, and the like. The three data preprocessing methods mentioned above are specifically implemented as follows:
(1) all data are normalized to the envelope magnitude maximum.
(2) Downsampling all data to D in the distance dimension down 。
(3) Upsampling all data to F in frame order dimension up . (when a specific platform is used in the method F up = 30). Namely, the data size normalization is realized by two steps (2) and (3).
(5) The LSTM network portion used to extract features may freely adjust network parameters based on the size of the envelope data, the signal characteristics of the radar, and other factors. Some of the parameters of the LSTM network are defined as follows:
(1) the input _ size of the input data is H in ＝D down I.e. the length of the input data in the distance dimension;
(2) input time series length L in ＝F up Dividing the envelope data into time sequences according to a frame sequence and inputting the time sequences into a recurrent neural network;
(3) hidden layer feature number (hidden _ size) is H cell The characteristic number of the matrix data output by the network can be regarded as the distance dimension length of the output data, and the parameter is adjusted according to the actual performance of the network;
(4) the characteristic number of the output data is H out ＝H cell ；
(5) RNN layers (num _ layers) of D R ；
(6) The group size (batch _ size) is N B ；
(7) Using a unidirectional LSTM network;
(8) the data output by the network is a matrix with the size of (N) B ,L in ,H out ). Other parameters can be set and adjusted automatically according to requirements.
(6) The Full Connection (FC) layer network portion used for classification is not of a particular form. The classifier of the FC layer maps the data output by the LSTM network to a one-dimensional tensor G 1×M And M is the number of gesture types needing to be recognized. G 1×M Each value of the tensor corresponds to the probability g of the envelope data discriminating a gesture of a certain type i Namely, the judgment result is:
wherein i is the number of the identified gesture.
The identification method of the invention is mainly characterized in that an LSTM network model and a training model are built in Python, and the method comprises the following specific steps:
(1) Reading the mat file under each gesture folder in sequence, storing all data of the training set and the test set as a data set in the forms of [ 'data', 'label' ], conveniently establishing the data set by using the succession of dataset classes provided by a Pythroch library, and storing the data in a Tensor (Tensor) form for convenient subsequent direct use.
(2) Reading a sensor of gesture data from the training set, reading a data part, namely a data part, wherein the data part is recorded as A, A is also a sensor data, and the size of the sensor data is [1, f, D ] 0 ]. The following data preprocessing is carried out on the raw materials in sequence:
(1) normalizing A according to the maximum value of the envelope amplitude to obtain data
That is, each element in the matrix A is divided by the maximum value of the element in A to obtain the normalized matrix
(2) Will be provided with
Down-sampling to D in the distance dimension down =100, the default function interpole of the torch.nn.functional library of Pytorch is used. Namely, it is
Is changed into
(3) Will be provided with
Up-sampling to F in frame order dimension up =30, the default function interpolate of the torch.nn.functional library of Pytorch is used. Namely, it is
Is changed into
That is, the data preprocessing is realized by the steps, and the preprocessed data is recorded as
(3) And establishing an LSTM model for feature extraction. LSTM network store. Nn. LSTM encapsulated using pytorech library. The structure of the network is shown in fig. 2, and the part inside the large rounded rectangle is a circular unit, namely a Cell. Each frame data of the pre-processed data a can be represented as a time series X (t, τ) = { X = { (X) } 1 (t),x 2 (t),...,x τ (t)|t∈[1,T]Given learning data, the development length thereof is τ, that is, τ is the number of features (that is, the number of distance units of a is 100); t is time-step (time-step), i.e. the direction of evolution of the sequence; t is the length of the time dimension of the sequence (i.e., the number of frames of A, 30). For two-dimensional envelope data organized in frames, it can be considered that τ is the total number of distance units per frame, T is the frame order, and T is the total number of frames. In fig. 2, three sigma gates are a forgetting gate, an input gate and an output gate respectively. The input gate determines the weight of X (t) and h (t-1) input to c (t); the forgetting gate determines the weight of c (t-1) iteration to c (t); the output gate determines the weight of the c (t) output to h (t). The output state is also a hidden layer state and will be used as an input of the next iteration repetition. The specific operation represented by each gate in the cyclic unit structure and the operation of the whole cyclic unit on the input X (t) to obtain the outputs h (t) and c (t) are as follows:
h(t)＝g o (t)·f h (c(t))
c(t)＝g f (t)·c(t-1)+g i (t)·f c (vh(t-1)+uX(t)+b)
g i (t)＝Sigmoid(v i h(t-1)+u i X(t)+b i )
g f (t)＝Sigmoid(v f h(t-1)+u f X(t)+b f )
g o (t)＝Sigmoid(v o h(t-1)+u o X(t)+b o )
where h (t) is the hidden layer state (hidden state),
the internal state (cell state) of the system at the previous time, o (t) is the output state. g (t) is a gating function, subscripts i, f, o respectively represent input gates, forgetting gates, output gates, and are Sigmoid functions (represented by σ) in the figure. f. of h (. Cndot.) and f c (. Cndot.) represents the activation functions of h (t) and c (t), respectively, both of which are Tanh functions. v and u are weight coefficients, and b is an offset.
The parameters of the part of the torch.nn.lstm function used are defined as follows:
(1) input _ size is set to 100, and this parameter is the characteristic number of the input data, i.e., the length τ of the input data in the distance dimension, and is recorded as H in ＝D down ＝100；
(2) Input time series length is noted as L in ＝F up =30, namely dividing the envelope data into time series according to the frame sequence and inputting the time series into the sequence length T of the LSTM network, wherein the parameter is determined by the input data and does not need to be set independently;
(3) the hidden _ size is set to 20, and this parameter is the hidden layer feature number, denoted as H cell =20, the length of the network output h (t) in the distance dimension, this parameter being adjusted according to the actual behaviour of the network;
(4) the characteristic number of the output data is recorded as H out ＝H cell =20, i.e. the length of o (t) of the network output in the distance dimension, which parameter is determined by the hidden _ size, without being set separately;
(5) num _ layers is set to 2, and the parameter is the number of LSTM network layers, denoted as D R ＝2；
(6) The batch _ size is set to 1, and the parameter is the size of the data set, denoted as N B ＝1；
(7) bidirectionality is set to False, and this parameter indicates that a unidirectional LSTM network is used;
other parameters can be set and adjusted according to needs.
The data output by the network is the sensor,size of (N) B ,L in ,H out ) Let the sensor data output by the LSTM layer be B and the size be B 1×30×20 。
(4) An FC layer network for classification is established as in fig. 3. The input is B, the output is one-dimensional Tensor and is marked as O 1×M M =5 is the kind of gesture to be recognized, including 4 basic gestures and invalid gestures designed. The network consists of the following components from top to bottom:
(1) a Flatten layer for flattening data, flattening the input data B into a one-dimensional Tensor with the length of N B ×L in ×H out ＝1×30×20＝600。
(2) Dropout layer, discarding the data of the previous layer with a probability of 0.5 by default.
(3) FC layer, single fully connected layer, for mapping data to gesture space O 1×M 。
(5) O is separated by softmax layer 1×M Mapping to gesture probability space G with zero elements 1×M And M is the number of gesture types needing to be recognized. G 1×M Each value of the tensor corresponds to the probability of distinguishing the envelope data as a certain type of gesture, namely the distinguishing result is as follows:
wherein i is the number of the identified gesture.
(6) And comparing the discrimination result i with 'label' of the data, namely the label, calculating a loss value through a cross entropy loss function, and feeding back.
(7) And (4) repeating the steps (2) to (6) until each data in the training set is added into the network.
(8) And (7) repeating the step 200 for times.
And completing the training of the model after the steps are completed. FIG. 3 is a block diagram of the LSTM network trained herein, cell is the LSTM loop unit, flatten is the data flattening layer, dropout is the discard layer, full Connect is the Full-connection network layer, and Softmax is the normalization classification layer. The trained model can be directly read, and the data of the test set is read for model testing. The identification accuracy of the model based on the LSTM time sequence obtained by the identification method of the invention on the data set can reach 93 percent.
Claims (8)
1. A millimeter wave radar gesture recognition method based on an envelope data time sequence is characterized by comprising the following steps:
step S1: acquiring data of different gestures by using a millimeter wave radar to form envelope data, wherein the envelope data of each gesture is two-dimensional matrix data in a frame sequence-distance unit organization form;
step S2: preprocessing the envelope data read in the step S1;
and step S3: putting the data preprocessed in the step S2 into an LSTM network for feature extraction;
and step S4: and obtaining a gesture recognition result through a classifier of a full connection layer.
2. The envelope data time sequence-based millimeter wave radar gesture recognition method of claim 1, wherein the millimeter wave radar is connected with a PC through a USB interface, and receives data from an upper computer program using radar data and stores the data; the USB-A radar platform of serial port communication is used during gesture datase:Sub>A collection, and the USB-A platform of serial port communication and the USB-C platform based on the HID communication protocol are used during testing.
3. The envelope data time sequence-based millimeter wave radar gesture recognition method of claim 1, wherein the frame length f of each envelope data is between 10 and 30 frames, and the acquisition time equivalent to each gesture is 2 to 3 seconds at the data transmission rate of serial port communication; the number of distance units per data is D 0 (ii) a And after the data are collected by the upper computer program, the data are packed and stored into a txt or mat format for subsequent processing.
4. The millimeter wave radar gesture recognition method based on the envelope data time sequence of claim 1, wherein a hand is placed right in front of a millimeter wave radar when gesture data is collected, and the radar is placed on a table or held in the palm; corresponding gesture actions are made within a certain time, pulse radar signals are reflected by hands, received by the receiving antenna and preprocessed in the radar chip, and then generated envelope data are directly transmitted back to the PC end and are packaged and stored through an upper computer program.
5. The envelope data time-series based millimeter wave radar gesture recognition method of claim 1, wherein the millimeter wave radar is any single-shot single-receive PCR radar or extremely narrow pulse radar.
6. The millimeter wave radar gesture recognition method based on the envelope data time sequence according to claim 1, wherein the preprocessing includes, but is not limited to, distance dimension down-sampling, normalization, and size unification, and the three preprocessing methods are specifically implemented as follows:
(1) normalizing all data according to the maximum value of the envelope amplitude;
(2) downsampling all data to D in the distance dimension down ；
(3) Upsampling all data to F in frame order dimension up Namely, the data size normalization is realized by two steps (2) and (3).
7. The envelope data time-series based millimeter wave radar gesture recognition method of claim 6, wherein the LSTM network part for extracting features freely adjusts network parameters according to the size of the envelope data and the signal characteristics of the radar, and the parameters of the LSTM network part are defined as follows:
the characteristic number of the input data is H in ＝D down I.e. the length of the input data in the distance dimension; input time series length L in ＝F up Dividing the envelope data into time sequences according to a frame sequence and inputting the time sequences into a recurrent neural network; the number of hidden layer features is H cell The characteristic number of the matrix data output by the network is regarded as the distance dimension length of the output data, and the parameter is adjusted according to the actual performance of the network; characteristics of the output dataNumber H out ＝H cell (ii) a RNN layer number D R (ii) a Group size N B (ii) a Using a unidirectional LSTM network; the data output by the network is a matrix with the size of (N) B ,L in ,H out ) (ii) a And other parameters are set and adjusted automatically according to needs.
8. The envelope data time-series based millimeter wave radar gesture recognition method of claim 7, wherein the full connection layer network part for classification has no specific form, an active layer, a BN layer and a Dropout layer are used for improving network performance, and a classifier of the full connection layer maps data output by the LSTM network to a one-dimensional tensor G 1×M Wherein M is the number of gesture types needing to be recognized; g 1×M Each value of the tensor corresponds to the probability g of the envelope data discriminating a certain type of gesture i Namely, the judgment result is:
wherein i is the number of the identified gesture.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211312638.XA CN115754956A (en) | 2022-10-25 | 2022-10-25 | Millimeter wave radar gesture recognition method based on envelope data time sequence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211312638.XA CN115754956A (en) | 2022-10-25 | 2022-10-25 | Millimeter wave radar gesture recognition method based on envelope data time sequence |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115754956A true CN115754956A (en) | 2023-03-07 |
Family
ID=85353632
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211312638.XA Pending CN115754956A (en) | 2022-10-25 | 2022-10-25 | Millimeter wave radar gesture recognition method based on envelope data time sequence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115754956A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116842457A (en) * | 2023-07-17 | 2023-10-03 | 中国船舶集团有限公司第七二三研究所 | Long-short-term memory network-based radar radiation source individual identification method |
-
2022
- 2022-10-25 CN CN202211312638.XA patent/CN115754956A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116842457A (en) * | 2023-07-17 | 2023-10-03 | 中国船舶集团有限公司第七二三研究所 | Long-short-term memory network-based radar radiation source individual identification method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111160139B (en) | Electrocardiosignal processing method and device and terminal equipment | |
| CN107024685A (en) | A kind of gesture identification method based on apart from velocity characteristic | |
| CN111178331B (en) | Radar image recognition system, method, apparatus, and computer-readable storage medium | |
| CN111738060A (en) | Human gait recognition system based on millimeter wave radar | |
| CN110007366A (en) | A kind of life searching method and system based on Multi-sensor Fusion | |
| WO2023029390A1 (en) | Millimeter wave radar-based gesture detection and recognition method | |
| CN111427031A (en) | Identity and gesture recognition method based on radar signals | |
| CN111950437A (en) | Gait recognition method and device based on deep learning model and computer equipment | |
| Kim et al. | Radar-based human activity recognition combining range–time–Doppler maps and range-distributed-convolutional neural networks | |
| CN115754956A (en) | Millimeter wave radar gesture recognition method based on envelope data time sequence | |
| Ullmann et al. | A survey on radar-based continuous human activity recognition | |
| CN116184394A (en) | Millimeter wave radar gesture recognition method and system based on multi-domain spectrogram and multi-resolution fusion | |
| CN110084209A (en) | A kind of real-time gesture identification method based on father and son's classifier | |
| Li et al. | Recognizing strawberry appearance quality using different combinations of deep feature and classifiers | |
| CN112826483B (en) | Fingertip video-based heart rate detection method, system and device | |
| CN110008847A (en) | A kind of stroke recognition methods based on convolutional neural networks | |
| CN112380903B (en) | Human body activity recognition method based on WiFi-CSI signal enhancement | |
| CN109034093A (en) | A kind of design and realization of quick dynamic Sign Language Recognition algorithm | |
| CN116794602A (en) | Millimeter wave radar dynamic gesture recognition method applied to interference environment | |
| CN110309689B (en) | Gabor domain gesture recognition detection method based on ultra-wideband radar | |
| CN114511873B (en) | Static gesture recognition method and device based on millimeter wave radar imaging | |
| CN113238222B (en) | Human body action recognition method based on envelope density characteristics | |
| CN117312935A (en) | Action category identification method, device, computer equipment and storage medium | |
| Quanyu | Pulse signal analysis based on deep learning network | |
| CN114387672A (en) | Human body behavior classification method based on time-space-frequency three-dimensional radar point cloud |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |