CN115079118A - Millimeter wave radar gait recognition method based on retrieval task - Google Patents
Millimeter wave radar gait recognition method based on retrieval task Download PDFInfo
- Publication number
- CN115079118A CN115079118A CN202210641204.8A CN202210641204A CN115079118A CN 115079118 A CN115079118 A CN 115079118A CN 202210641204 A CN202210641204 A CN 202210641204A CN 115079118 A CN115079118 A CN 115079118A
- Authority
- CN
- China
- Prior art keywords
- time
- frequency
- signal
- gait recognition
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/02—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00
- G01S7/41—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S13/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/02—Systems using reflection of radio waves, e.g. primary radar systems; Analogous systems
Abstract
The invention relates to a millimeter wave radar gait recognition method based on a retrieval task, which comprises the following steps: mixing the echo signal with the transmitting signal to obtain a difference frequency signal in a complex exponential form, and performing discrete time sampling processing on the difference frequency signal to obtain a discrete time signal; carrying out incoherent accumulation on the difference frequency signal in the form of a discrete time signal to obtain a signal after incoherent accumulation, carrying out distance dimension discrete Fourier transform on the signal after incoherent accumulation to obtain a time-distance graph, and carrying out short-time Fourier transform on the time-distance graph to obtain a time-frequency graph; constructing a first data set based on the time-frequency diagram after the nonlinear normalization operation; training a gait recognition network model by using a first data set to obtain a trained gait recognition network model; and inputting the time-frequency diagram to be recognized into the trained gait recognition network model to obtain a recognition result. The invention aims to provide a complete scheme for completing a retrieval task in gait recognition by using a time-frequency graph, and the practical application scene of a gait recognition model is enlarged.
Description
Technical Field
The invention belongs to the technical field of identification, and relates to a millimeter wave radar gait identification method based on a retrieval task.
Background
Gait recognition is a new biological feature recognition technology, aims to identify the identity through the walking posture of people, and has the advantages of non-contact remote distance and difficulty in disguising compared with other biological recognition technologies, so that the gait recognition technology has wide application in the fields of safety protection, medical treatment, terrorism and riot prevention, intelligent video monitoring and the like. Gait recognition can be divided into a classification task and a retrieval task, for the classification task, the identity of a sample in a test set must appear in a training set, otherwise, a classifier cannot correctly judge the identity of the test sample, and the application of radar gait recognition in an actual scene is limited. The search task is to give a query sample, search out a sample with the highest similarity from the sample library according to a certain similarity index, and take the identity of the searched sample as the identity of the query sample.
Common sensors for acquiring gait data include optical sensors, wearable devices, radars, etc., of which optical sensors and radars are most widely used. Compared with an optical sensor, the millimeter wave radar has the advantages of being free of influence of light intensity, protecting personal privacy and the like, has high distance, speed resolution and certain angle resolution, and is easy to capture human body micro-motion information, so that the gait recognition method based on the millimeter wave radar gradually receives wide attention. Because the motion mode and the electromagnetic scattering property of each person are different, the micro-motion information generated by the swinging of the limbs and other parts when the person walks can be used as the unique characteristic of the person. The time-frequency graph obtained by short-time Fourier transform of the radar echo signals can reflect the rich micro-motion information of the target, so that the gait recognition based on the time-frequency graph data of the millimeter wave radar has important research significance.
First, the existing gait recognition technology is directed to classification tasks, and for a traditional classification model, when a query sample with no identity appears in a training set during testing, the model cannot correctly judge the identity of the traditional classification model. The most common solution is to retrain the classification model, which is difficult to implement in practice: on one hand, the classification model can not be retrained by taking sufficient training data of the query sample; on the other hand, even if there is sufficient data to retrain the model, the time and computational cost are very large, which severely limits the practical application scenarios of the technology.
Secondly, a sample library constructed by the existing gait recognition technology is not perfect enough, most technical means aim at the situation that a human body walks along the radial direction of a radar sight line, the situation that the human body walks at a large angle with the radar sight line or walks along the tangential direction of the radar sight line, the situation that the human body walks by wearing overcoat and a backpack and the like is not considered, and a model trained by the sample based on a simple walking condition cannot be suitable for other walking situations.
Therefore, how to construct a complete system of millimeter wave radar gait data acquisition, signal processing and identity recognition provides a possible problem to be solved for realizing gait recognition in real time.
Disclosure of Invention
In order to solve the above problems in the prior art, the present invention provides a wideband input buffer for high-speed high-precision analog-to-digital converter. The technical problem to be solved by the invention is realized by the following technical scheme:
the embodiment of the invention provides a millimeter wave radar gait recognition method based on a retrieval task, which comprises the following steps:
step 1, mixing an echo signal acquired by a millimeter wave radar with a transmitting signal to obtain a difference frequency signal in a complex exponential form, and performing discrete time sampling processing on the difference frequency signal in the complex exponential form to obtain a difference frequency signal in a discrete time signal form;
step 2, performing incoherent accumulation on the difference frequency signal in the form of the discrete time signal to obtain a signal after incoherent accumulation, performing distance dimension discrete Fourier transform on the signal after incoherent accumulation to obtain a time-distance graph, and performing short-time Fourier transform on the time-distance graph to obtain a time-frequency graph;
step 3, constructing a first data set based on the time-frequency diagram after the nonlinear normalization operation, wherein the first data set comprises samples which are worn normally, dressed and worn with a satchel;
step 4, training a gait recognition network model by using the first data set to obtain the trained gait recognition network model, wherein the gait recognition network model comprises ResNet18 which does not contain a first maximum pooling layer and a time-frequency graph time dimension feature extraction module;
and 5, inputting the time-frequency diagram to be recognized into the trained gait recognition network model to obtain a recognition result.
In one embodiment of the present invention, the step 1 comprises:
step 1.1, obtaining the result T x The transmitting antenna periodically transmits linear frequency modulation continuous waves to the human body to be reflected by the human body x Echo signals received by the receiving antennas;
step 1.2, adding T x ×R x The echo signals of each channel and the corresponding transmitting signals are subjected to frequency mixing through an orthogonal frequency mixer, and the signals after frequency mixing are subjected to low-pass filtering processing to obtain difference frequency signals in the form of complex exponentials;
and step 1.3, performing discrete time sampling processing on the difference frequency signal in the complex exponential form to obtain the difference frequency signal in the discrete time signal form.
In one embodiment of the present invention, the difference frequency signal in complex exponential form is:
wherein, the first and the second end of the pipe are connected with each other,
is the difference frequency signal of the ith channel in complex exponential form,
for the echo signal received by the ith channel,
is composed of
Corresponding to the transmitted signal, the LPF is a low pass filter,
is the frequency of the difference frequency signal,
the phase of the difference frequency signal is A, and the amplitude of the difference frequency signal is A;
the difference frequency signal in the form of the discrete-time signal is:
wherein the content of the first and second substances,
the difference signal being in the form of a discrete-time signal of the ith channel, N s In order to sample the number of points in a discrete time,
for discrete time angular frequency, F s Is a discrete time sampling frequency.
In one embodiment of the present invention, the step 2 comprises:
step 2.1, mixing T x ×R x Carrying out incoherent accumulation on the difference frequency signals in the form of discrete time signals of each channel to obtain signals subjected to incoherent accumulation;
step 2.2, performing discrete Fourier transform on the non-coherent accumulated signals along a distance dimension to obtain the time and distance graph;
step 2.3, summing the time and distance graphs according to columns and averaging to obtain a reference vector;
2.4, subtracting the reference vector from each time dimension of the time and distance graph to obtain a time and distance graph after static clutter is filtered;
step 2.5, summing the time after the static noise is filtered and the distance unit where the human body is located in the distance graph, and performing short-time Fourier transform on the vector obtained by summing along a time dimension to obtain a time-frequency graph, wherein the time-frequency graph is as follows:
wherein, X 0 (t, f) is a time-frequency diagram, w (t) is a window function, t is time, f is frequency, m is a sampling point in the slow time dimension, N 0 The total number of distance units occupied by the human body.
In one embodiment of the present invention, the step 3 comprises:
step 3.1, carrying out nonlinear normalization operation on the time-frequency diagram to obtain a time-frequency diagram of the nonlinear normalization operation, wherein the time-frequency diagram of the nonlinear normalization operation of P persons is total;
and 3.2, taking the time-frequency diagram of at least one part of linear normalization operation in the P human bodies as data in the first data set.
In one embodiment of the present invention, the step 4 comprises:
step 4.1, inputting a reference sample, a positive example sample and a negative example sample in the training sample set into the gait recognition network model, wherein the reference sample and the positive example sample are different samples of the same human body, and the reference sample and the negative example sample are different samples of different human bodies;
step 4.2, processing the ternary loss function of the gait recognition network model by using a random gradient descent algorithm to minimize the ternary loss function;
and 4.3, acquiring a trained gait recognition network model through the minimum ternary loss function.
In an embodiment of the present invention, the ResNet18 includes four residual modules connected in sequence, where a feature map output by a previous residual module is input to a next residual module, and a feature map output by each residual module is input to the time-frequency graph time dimension feature extraction module, the time-frequency graph time dimension feature extraction module correspondingly outputs a plurality of first feature vectors, all the first feature vectors corresponding to all the residual modules are subjected to dimension transformation by using a full connection layer to obtain a plurality of second feature vectors, and all the second feature vectors are subjected to end-to-end splicing to obtain 1 gait feature vector.
In an embodiment of the present invention, the processing method of the time-frequency graph time dimension feature extraction module includes:
s1.1, dividing a feature map with the dimension of C multiplied by H multiplied by W into N parts along the width direction to obtain N parts of sub-feature maps, wherein C is the number of channels, H is the height of the feature map, and W is the width of the feature map;
s1.2, for each sub-feature diagram
Using global maximum pooling in dimensionality, wherein each sub-feature map becomes a sub-feature vector with the dimensionality of C, and the feature map becomes N third feature vectors with the dimensionality of C;
s1.3, averaging the N third feature vectors with the dimensionality of C to obtain 1 first feature vector with the dimensionality of C.
In an embodiment of the present invention, after step 4, further comprising:
inputting all samples in a sample library into a trained gait recognition network model to obtain a first step state feature vector, wherein the samples in the sample library are all normal samples;
inputting one sample in query samples into a trained gait recognition network model to obtain a second-step state feature vector, wherein the query samples comprise samples worn normally, dressed in overcoat and worn in a satchel;
sequentially calculating Euclidean distances between the second step state feature vector and the first step state feature vectors of all samples in the sample library;
selecting the sample in the sample library with the minimum Euclidean distance.
In one embodiment of the present invention, the step 5 comprises:
step 5.1, inputting a time-frequency diagram to be recognized into a trained gait recognition network model to obtain a gait feature vector;
step 5.2, sequentially calculating Euclidean distances between the gait feature vector and gait feature vectors of all samples in a preset sample library;
and 5.3, selecting the sample in the preset sample library with the minimum Euclidean distance as a final identification result.
Compared with the prior art, the invention has the beneficial effects that:
firstly, the invention provides a complete scheme for completing retrieval tasks in gait recognition by using time-frequency diagram data for the first time, when a person with a new identity appears, a recognition model does not need to be retrained, only a query sample needs to be input into the trained gait recognition model, and the identity of the query sample is accurately retrieved from a sample library in a similarity measurement mode. The method solves the problem that the classification task cannot correctly judge the identity which does not appear in the training set, and has wider application scenes compared with the traditional classification task, such as key personnel monitoring, access control systems and the like.
Secondly, the invention perfects the whole process of gait recognition from the acquisition of original echo of the target radar, signal processing to characteristic extraction. A relatively complete time-frequency image sample library is constructed, the samples comprise samples of human bodies walking along various angles of the radar and different wearing conditions, and a gait recognition model is trained based on the samples. The gait recognition method integrates the whole process of acquiring gait recognition from the original echo, designs the gait recognition model aiming at multiple walking angles and cross-wearing conditions, and improves the generalization of the model to a certain extent.
Other aspects and features of the present invention will become apparent from the following detailed description, which proceeds with reference to the accompanying drawings. It is to be understood, however, that the drawings are designed solely for purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims. It should be further understood that the drawings are not necessarily drawn to scale and that, unless otherwise indicated, they are merely intended to conceptually illustrate the structures and procedures described herein.
Drawings
Fig. 1 is a schematic flowchart of a millimeter wave radar gait recognition method based on a retrieval task according to an embodiment of the present invention;
FIGS. 2a and 2b are time-frequency diagrams of an unfiltered static clutter and a filtered static clutter according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a gait recognition network model according to an embodiment of the invention;
fig. 4 is a schematic diagram of a time dimension feature extraction module for a time-frequency graph according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to specific examples, but the embodiments of the present invention are not limited thereto.
Example one
Referring to fig. 1, fig. 1 is a schematic flow chart of a millimeter wave radar gait recognition method based on a retrieval task according to an embodiment of the present invention, and the present invention provides a millimeter wave radar gait recognition method based on a retrieval task, where the millimeter wave radar gait recognition method includes:
step 1, mixing an echo signal acquired by a millimeter wave radar with a transmitting signal to obtain a difference frequency signal in a complex exponential form, and performing discrete time sampling processing on the difference frequency signal in the complex exponential form to obtain the difference frequency signal in a discrete time signal form.
The millimeter wave radar transmits a linear frequency modulation continuous wave composed of continuous linear frequency modulation signals, and the radar configuration parameters mainly comprise the number T of transmitting antennas x The number of receiving antennas R x Carrier frequency ofRate f c Duration of chirp τ, chirp rate S, ADC sample rate f s Number of sampling points N for each chirp signal s The repetition period T of the chirp signal c The number N of chirp signals contained in each frame c Total number of frames N f Observation time is T c ×N c ×N f . The parameters are determined by performance indexes such as distance resolution, unambiguous distance, speed resolution, unambiguous speed and the like required in an actual measurement scene.
In one embodiment, step 1 may comprise:
step 1.1, obtaining the result T x The transmitting antenna periodically transmits linear frequency modulation continuous waves to the human body to be reflected by the human body x Echo signals received by the receiving antennas.
In particular, in an indoor scenario, by T x The transmitting antenna periodically transmits linear frequency-modulated continuous waves which are reflected by a walking human body and then are transmitted by the R x Receiving echo signals by a receiving antenna, wherein each acquisition sample has T x ×R x Echo data of each channel. The human body walking angle is as follows: the angle is 0 degree, 30 degree, 45 degree, 60 degree, 90 degree, 30 degree, 45 degree and 60 degree with the radial line of sight of the radar; the wearing conditions of the human body during walking are as follows: normally, wearing overcoat and messenger bags, wherein the wearing of overcoat can shield the movement of the legs, the wearing of messenger bags can shield the movement of the arms, the wearing of messenger bags can not affect the movement of the legs and the arms, the wearing of messenger bags normally can not shield the legs, and the two arms can swing; the total number of the collected people is P. A total of 10 samples were taken from each person (6 normal wear, 2 overcoat wear, 2 satchel) each containing 8 walking angle samples, i.e. a total of 80 samples were taken from each person.
Step 1.2, mixing T x ×R x The echo signals of each channel and the corresponding transmitting signals are subjected to frequency mixing through an orthogonal frequency mixer, the frequency-mixed signals are subjected to low-pass filtering processing to obtain difference frequency signals in a complex exponential form, and the difference frequency signals in the complex exponential form are as follows:
wherein the content of the first and second substances,
is the difference frequency signal of the ith channel in complex exponential form,
for the echo signal received by the ith channel,
is composed of
Corresponding to the transmitted signal, the LPF is a low pass filter,
is the frequency of the difference frequency signal,
the phase of the difference signal and a the amplitude of the difference signal.
And step 1.3, performing discrete time sampling processing on the difference frequency signal in the form of complex exponential to obtain the difference frequency signal in the form of the discrete time signal.
Specifically, for T x ×R x All the channel data are subjected to the operation to obtain T x ×R x A difference frequency signal in the form of a complex exponential for each channel. The complex exponential type difference frequency signal obtained at this time is in the form of an analog signal, and in order to facilitate the subsequent digital signal processing flow, discrete time sampling is performed on the complex exponential type difference frequency signal, the complex exponential type difference frequency signal is converted into a discrete time signal form and is stored, and the discrete time signal type difference frequency signal is as follows:
wherein the content of the first and second substances,
the difference signal being in the form of a discrete-time signal of the ith channel, N s In order to sample the number of points in a discrete time,
for discrete time angular frequency, F s Is a discrete time sampling frequency.
And 2, performing incoherent accumulation on the difference frequency signal in the form of the discrete time signal to obtain a signal after incoherent accumulation, performing distance dimension discrete Fourier transform on the signal after incoherent accumulation to obtain a time-distance graph, and performing short-time Fourier transform on the time-distance graph to obtain a time-frequency graph.
In one embodiment, step 2 may comprise:
step 2.1, adding T x ×R x Carrying out incoherent accumulation on the difference frequency signals in the form of discrete time signals of each channel to obtain signals after incoherent accumulation, wherein the signals after incoherent accumulation are as follows:
wherein x is IF [n]Is a non-coherently accumulated signal.
And 2.2, performing discrete Fourier transform on the signal after incoherent accumulation along a distance dimension to obtain a time-distance graph.
Specifically, discrete fourier transform is performed on the signal after incoherent accumulation along a distance dimension (fast time dimension) to obtain a time-distance graph, the process is equivalent to windowing, the signal-to-noise ratio of the signal can be effectively improved, and the time-distance graph is as follows:
S(n,m)＝DFT{x IF [n]}
wherein S (n, m) is a time and distance graph, DFT is discrete time Fourier change, n is a fast time dimension sampling point, and m is a slow time dimension sampling point.
Step 2.3, summing the time and distance graphs according to columns and averaging to obtain a reference vector, wherein the reference vector is as follows:
where M represents the total number of samples in the slow time dimension.
In this embodiment, because the existence of static clutter, can produce very strong clutter peak in frequency zero department, again because human walking speed is slower, human frequency component distributes around zero-frequency, and this has caused target and clutter in zero-frequency department to produce serious aliasing, influences follow-up characteristic extraction and discernment, consequently needs to filter static clutter in the echo. In the embodiment, a vector mean phase elimination method is adopted to filter out static clutter, and the algorithm principle is as follows: if the echo signals are static targets, the phases of different echo signals are the same, the average value after vector summation and accumulation is large, and the amplitude of each echo signal after the average value is reduced becomes very small; if the target is a moving target or a micro-moving target, the phase of each echo signal is different due to the movement of the target, the average value after vector summation and accumulation is small, and the influence on the signal amplitude is small after the average value of each signal is reduced. Therefore, after static clutter suppression, the relative amplitude of motion and the micro-motion target becomes large. If the method for filtering out the static clutter is better, the corresponding replacement can be made. The present embodiment uses, but is not limited to, vector mean phase elimination, which is an algorithm used to filter out static clutter in a time-distance map.
In addition, the mode of filtering out static clutter can be replaced by a moving target display MTI, and an MTI filter utilizes the difference of doppler frequencies of clutter and a moving target, so that the frequency response of the filter has a deeper stop band at integral multiples of direct current and pulse repetition frequencies, and the suppression at other frequency points is weaker, so that a static target and the static clutter are suppressed through a deeper 'notch', and a one-time canceller is commonly used in practice:
y(n)＝x(n)-x(n-1)
wherein, x (n) represents the nth pulse, x (n-1) represents the nth pulse, and y (n) represents the result of the nth pulse for filtering out the static clutter.
Step 2.4, subtracting the reference vector from each time dimension of the time-distance map to obtain a time-distance map after static clutter is filtered, wherein the time-distance map after static clutter is filtered is as follows:
S'(n,m)＝S(n,m)-C(n)
wherein, S' (n, m) is a time and distance graph after static clutter is filtered.
Step 2.5, summing the time after the static noise is filtered and the distance unit where the human body is located in the distance graph, and performing short-time Fourier transform on the vector obtained by summing along a time dimension to obtain a time-frequency graph, wherein the time-frequency graph is as follows:
wherein, X 0 (t, f) is a time-frequency diagram, w (t) is a window function, t is time, f is frequency, m is a sampling point in the slow time dimension, N 0 On the distance-time graph, the rows represent distances, and each row represents a distance cell.
In this embodiment, the effect of using vector mean phase elimination to filter out the static clutter is shown in fig. 2a and fig. 2b, where fig. 2a shows a time-frequency diagram without filtering out the static clutter and fig. 2b shows a time-frequency diagram with filtering out the static clutter. It can be seen that spurious peaks around frequency 0 are effectively suppressed.
And 3, constructing a first data set based on the time-frequency diagram after the nonlinear normalization operation, wherein the first data set comprises samples which are worn normally, dressed and worn with a satchel.
On one hand, the radar sensor has high azimuth sensitivity, and the change of the time-frequency diagram can be caused by changing the angle between the radar sensor and the radar during walking; on the other hand, different wearing conditions of the human body during walking can change the RCS value of the human body and the body part reflecting electromagnetic waves, thereby changing the micro-motion information of the human body. The above problems affect the performance of gait recognition, so that a complete human gait data set needs to be constructed as much as possible to extract the robust gait features of human walking. The data set construction is implemented as follows:
step 3.1, performing nonlinear normalization operation on the time-frequency diagram to obtain a time-frequency diagram of the nonlinear normalization operation, wherein the time-frequency diagram of the nonlinear normalization operation of the person body is P in total, and the time-frequency diagram of the nonlinear normalization operation is as follows:
wherein, X is a time-frequency diagram of nonlinear normalization operation, mu is the mean value of all elements of a single time-frequency diagram, and sigma is the standard deviation of all elements of the single time-frequency diagram.
In the embodiment, the gait recognition model based on the retrieval task is designed, and in order to ensure that the identities of training and testing samples are not coincident, the collected P persons are the P persons before and before P persons train Human samples as training data, P remaining test (i.e., P-P) train ) Human samples were used as test samples.
And 3.2, taking the time-frequency diagram of at least one part of linear normalization operation in the P human bodies as data in the first data set.
Specifically, a training data set (i.e., a first data set) is constructed: front P train The human time-frequency pattern originally forms a training data set, and each person acquires 80 samples according to the data acquisition setting in the step 1.
Step 3.3, establishing a test data set: remaining P test The human time-frequency pattern book forms a test data set (i.e., a second data set), and as can be seen from the data acquisition setting in step 1, each person acquires 10 groups of data, and each group of data contains 8 samples at different walking angles. Dividing the test data set into a query sample set and a sample library, wherein the query sample set comprises 48 samples of 2 groups of normal wearing samples, 2 groups of overcoat wearing samples and 2 groups of shoulder-straps wearing samples; sample pool containing remainsThe remaining 4 groups were worn with normal specimens, for a total of 32.
And 4, training the gait recognition network model by using the first data set to obtain the trained gait recognition network model, wherein the gait recognition network model comprises ResNet18 which does not contain the first maximum pooling layer and a time-frequency graph time dimension feature extraction module.
Specifically, the present embodiment is directed to design a gait recognition network model for retrieval tasks of multiple walking angles and cross-wearing conditions, please refer to fig. 3, the gait recognition network model mainly includes: 1) the reference network based on ResNet18 mainly comprises four residual modules connected in sequence and used for extracting gait features of different levels in a time-frequency diagram; 2) and the time-frequency graph time dimension characteristic extraction module is used for extracting gait characteristics of human walking in different time intervals. The embodiment adopts a layered feature fusion strategy to fuse the low-layer features and the high-layer features of the network; in the embodiment, the distance between the samples of the same type is shortened and the distance between the samples of different types is lengthened in the feature space by utilizing the ternary loss constraint.
In one embodiment, step 4 may comprise:
and 4.1, inputting a reference sample, a positive sample and a negative sample in the training sample set into the gait recognition network model, wherein the reference sample and the positive sample are different samples of the same human body, and the reference sample and the negative sample are different samples of different human bodies.
In this embodiment, the residual network ResNet18 is used as a reference feature extraction network to extract gait features, and the residual structure makes information flow between layers easier, including feature reuse in forward propagation and mitigation of gradient signal disappearance in backward propagation. The time-frequency diagram size of the network input is 224 × 224. The first largest pooling layer of ResNet18 was removed, resulting in a feature size that was twice the size of the feature obtained for the original ResNet18 network.
In this embodiment, the processing method of the time-frequency graph time-dimension feature extraction module includes:
s1.1, dividing the feature map with the dimension of C multiplied by H multiplied by W into N parts along the width direction to obtain N parts of sub-feature maps, wherein C is the number of channels, H is the height of the feature map, and W is the width of the feature map.
S1.2, for each sub-feature map
And dimension using global maximum pooling, wherein each sub-feature map is changed into a sub-feature vector with dimension C, and the feature maps are changed into N third feature vectors with dimension C.
S1.3, averaging the N third feature vectors with the dimensionality of C to obtain 1 first feature vector with the dimensionality of C.
That is to say, the gait features extracted in different time intervals in the time-frequency diagram are different, the local features with fine granularity are extracted in a shorter time interval, and the global features with coarse granularity are extracted in the whole time interval. The local and global features are fully utilized to improve the identification performance of the network, so that the invention provides a time-frequency graph time dimension feature extraction module with the fusion of the local features and the global features as shown in fig. 4. The specific process comprises the following steps: 1) the feature map F having dimensions C × H × W is divided into N in the width direction. Dimension of feature map becomes
2) The height of the feature map represents the frequency dimension of the time-frequency map, and the width of the feature map represents the time dimension of the time-frequency map. To aggregate gait characteristics at shorter intervals, for each profile
Using global max pooling in dimension, the feature map becomes N feature vectors of dimension C. Among the reasons for using global max pooling instead of global average pooling are: the maximum response value on the time-frequency diagram is usually the point with most characteristic force in the time-frequency diagram, and the response value of other useless information (such as noise) is generally smaller. 3) And averaging the N characteristic vectors for the characteristic of the aggregation time dimension, and finally changing the characteristic diagram into 1 characteristic vector with the dimension of C. 4) In order to learn the gait characteristics in different time intervals, the invention sets N to be 1, 2 and 7, and respectively carries out the operations, and most preferablyFinally, 3 feature vectors with the dimensionality of C are obtained.
Therefore, the ResNet18 includes four sequentially connected residual error modules, where the feature map output by the previous residual error module is input to the next residual error module, the feature map output by each residual error module is input to the time-frequency map time dimension feature extraction module, the time-frequency map time dimension feature extraction module correspondingly outputs a plurality of first feature vectors (e.g., the time-frequency map time dimension feature extraction module correspondingly outputs 3 first feature vectors with dimension C when N is 1, 2, and 7), all the first feature vectors corresponding to all the residual error modules are subjected to dimension transformation by using the full connection layer to obtain a plurality of second feature vectors, and all the second feature vectors are subjected to end-to-end splicing to obtain 1 gait feature vector.
That is to say, the ResNet18 has four residual modules, the feature graph output by the shallow layer of the network shows the contour feature and the texture feature of the time-frequency graph, and the feature graph output by the deep layer of the network shows more semantic information of the time-frequency graph. In order to fully utilize the low-level features and the high-level features, the invention uses a strategy of hierarchical feature fusion, four groups of feature maps output by four residual modules use a time-frequency map time dimension feature extraction module to extract time dimension features, each group of feature maps obtain 3 feature vectors, and the four groups of feature maps obtain 12 feature vectors. And performing dimension transformation on the 12 feature vectors by using full connection with different parameters to obtain 12 1024-dimensional feature vectors, and finally splicing the 12 feature vectors end to obtain 1 12288-dimensional gait feature vector.
And 4.2, processing the ternary loss function of the gait recognition network model by using a random gradient descent algorithm so as to minimize the ternary loss function.
The calculation of the ternary loss function requires three samples to form a triplet, and the three samples are respectively a reference sample, a positive sample which is the same as the reference sample and a negative sample which is different from the reference sample. The purpose of optimizing the loss is to shorten the distance between the reference sample and the positive example sample and shorten the distance between the reference sample and the negative example sample, wherein the ternary loss function is:
L＝max(d(a,p)-d(a,n)+margin,0)
wherein d is the calculated Euclidean distance, a is a reference sample, p is a positive example sample, n is a negative example sample, and margin is a constant greater than 0.
In the network training process, the mth sample of the ith person needs to be randomly selected every time the parameters are updated iteratively
Randomly selecting the nth sample of the ith person as a reference sample
As positive example, randomly selecting the q sample of the jth person
As negative example samples, the above three samples are constructed into a triplet to use a ternary loss function.
And 4.3, acquiring the trained gait recognition network model through the minimum ternary loss function.
That is, when the ternary loss function is minimized, the gait recognition network model training is completed.
And 5, acquiring the identity of the query sample based on the gait recognition network model.
And 5.1, inputting all samples in the sample library into the trained gait recognition network model to obtain a first step state feature vector, wherein the samples in the sample library are all normal wearing samples.
And 5.2, inputting one sample in the query samples into the trained gait recognition network model to obtain a second step state feature vector, wherein the query samples comprise samples worn normally, dressed in overcoat and worn in satchel.
And 5.3, sequentially calculating Euclidean distances between the second-step state feature vector and the first-step state feature vectors of all samples in the sample library, wherein the Euclidean distances are as follows:
wherein d is the Euclidean distance, x is the second step state feature vector of the query sample, y is the first step state feature vector of the sample in the sample library, and n is the dimension of the feature vector;
and 5.4, selecting the sample in the sample library with the minimum Euclidean distance.
Specifically, the query sample and the euclidean distances calculated by all samples in the sample library are sorted, the sample in the sample library with the minimum euclidean distance is selected, and the identity of the sample in the sample library is the identity of the query sample.
The embodiment can verify the identification accuracy of the gait identification network model through the result obtained in the step 5.
And 6, inputting the time-frequency diagram to be recognized into the trained gait recognition network model to obtain a recognition result. And the time-frequency graph to be identified is the time-frequency graph corresponding to the person to be identified.
In one embodiment, step 6 may comprise:
step 6.1, inputting a time-frequency diagram to be recognized into a trained gait recognition network model to obtain a gait feature vector;
step 6.2, sequentially calculating Euclidean distances between the gait feature vectors and the gait feature vectors of all samples in a preset sample library;
and 6.3, selecting the sample in the preset sample library with the minimum Euclidean distance as a final identification result.
The preset sample library may be a sample library used for identification in actual application, and the samples included in the preset sample library may be set according to the actual application.
The effect of the present invention will be further explained by combining the actual measurement data experiment as follows:
1. experimental conditions and experimental contents
The software platform adopted in the experiment is as follows: windows10 operating system, Matlab R2021b, python3.6, pytorch.
The hardware platform adopted in this experiment is: dell T7910 workstation, CPU: intel Core (TM) i7-4770, GPU: NVIDIA GTX 1080Ti, radar millimeter wave radar TI Awr 1843.
The millimeter wave radar parameters adopted in the experiment are as follows: the carrier frequency is 77GHz, the frequency modulation bandwidth is 400MHz, the frequency modulation duration is 40.96us, the frequency modulation slope is 9.766MHz/us, the chirp signal repetition period is 78.125us, the idle time is 37.165us, the sampling frequency is 12.5Mhz, the number of sampling points of each chirp is 512, and the number of chirp signals per frame is 256;
the experimental data adopts an actually measured millimeter wave radar indoor human body walking data set, the data set comprises time-frequency pattern data of 116 normal wearing people, wearing overcoat and messenger bags, walking along the radial direction of the radar at 0 degree, 30 degrees, 45 degrees, 60 degrees, 90 degrees, 30 degrees, 45 degrees and 60 degrees, wherein the normal walking is 6 groups, wearing overcoat is 2 groups, and the messenger bag is 2 groups. The data set consisted of 9280 time-frequency plot samples, each time-frequency plot being 2.4 seconds long. The training set comprises a time-frequency graph of 74 persons, and 5920 samples are total; the test set comprises time-frequency graphs of the remaining 42 persons, and 3360 samples are totally included, wherein the query set comprises 2 groups of normal walking, 2 groups of overcoat wearing and 2 groups of inclined satchel, and 2016 samples are totally included; the sample library was normal walking 6 groups, for a total of 1344 samples. The recognition model is trained using training set samples, and testing is performed using the query set and samples in the sample library.
2. Results and analysis of the experiments
The invention aims to train a gait recognition model of cross-wearing conditions, namely, the wearing conditions in a sample library are normal, and concentrated wearing conditions are inquired to be normal, overcoat and cross-satchel. The results of the experiment are shown in table 1:
TABLE 1 Experimental results under three wearing conditions
As can be seen from table 1, the method provided by the present invention can achieve 94% of recognition accuracy without wearing, and the accuracy of wearing the overcoat and the messenger bag is reduced, because the walking mode of the human body is changed by wearing the overcoat and the messenger bag, which results in the change of the gait features extracted from the time-frequency diagram. The gait recognition method firstly proposes a gait recognition framework designed aiming at the retrieval task, enlarges the application scene of the gait recognition system in practice, and has higher practical application value.
The invention firstly provides a complete scheme for completing the retrieval task in gait recognition by using time-frequency image data, and when a person with a new identity appears, the identity of the query sample can be accurately retrieved from the sample library. The method is widely applied to actual scenes, such as monitoring of key personnel, access control systems and the like.
The invention perfects the whole process of gait recognition from the acquisition of the original echo of the target radar, signal processing to characteristic extraction. In particular, a vector mean phase elimination method is used for filtering static clutter in the signal processing process, and the method can keep more micro-motion information of the human body.
The gait recognition network provided by the invention extracts gait features in different time intervals by using a method of dividing the feature map along the time dimension; and a strategy of hierarchical feature fusion is used for fusing the contour feature and the texture feature of the time-frequency graph extracted from the lower layer of the network and the semantic feature extracted from the upper layer of the network, so that the representation of the gait feature is more complete.
The gait recognition method collects the gait data of different walking angles and different wearing conditions, and the multi-condition walking data is used in the model training process to improve the generalization of the gait recognition network to a certain extent, so that the collection mode is more in line with the actual condition.
The network structure parameter quantity provided by the invention is not large, and when a person with a new identity appears, the recognition model does not need to be retrained, so that the model is more suitable for edge calculation of an actual scene.
In the description of the invention, the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic data point described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristic data points described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples described in this specification can be combined and combined by those skilled in the art.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (10)
1. A millimeter wave radar gait recognition method based on retrieval tasks is characterized by comprising the following steps:
step 1, mixing an echo signal acquired by a millimeter wave radar with a transmitting signal to obtain a difference frequency signal in a complex exponential form, and performing discrete time sampling processing on the difference frequency signal in the complex exponential form to obtain a difference frequency signal in a discrete time signal form;
step 2, performing incoherent accumulation on the difference frequency signal in the form of the discrete time signal to obtain a signal after incoherent accumulation, performing distance dimension discrete Fourier transform on the signal after incoherent accumulation to obtain a time-distance graph, and performing short-time Fourier transform on the time-distance graph to obtain a time-frequency graph;
step 3, constructing a first data set based on the time-frequency diagram after the nonlinear normalization operation, wherein the first data set comprises samples which are worn normally, dressed and worn with a satchel;
step 4, training a gait recognition network model by using the first data set to obtain the trained gait recognition network model, wherein the gait recognition network model comprises ResNet18 which does not contain a first maximum pooling layer and a time-frequency graph time dimension feature extraction module;
and 5, inputting the time-frequency diagram to be recognized into the trained gait recognition network model to obtain a recognition result.
2. The millimeter wave radar gait recognition method based on the retrieval task, according to claim 1, wherein the step 1 comprises:
step 1.1, obtaining the result T x The transmitting antenna periodically transmits linear frequency modulation continuous waves to the human body to be reflected by the human body x Echo signals received by the receiving antennas;
step 1.2, adding T x ×R x The echo signals of each channel and the corresponding transmitting signals are subjected to frequency mixing through an orthogonal frequency mixer, and the signals after frequency mixing are subjected to low-pass filtering processing to obtain difference frequency signals in the form of complex exponentials;
and step 1.3, performing discrete time sampling processing on the difference frequency signal in the complex exponential form to obtain the difference frequency signal in the discrete time signal form.
3. The millimeter wave radar gait recognition method based on retrieval task of claim 2, wherein the difference frequency signal in the form of complex exponential is:
wherein the content of the first and second substances,
is the difference frequency signal of the ith channel in complex exponential form,
for the echo signal received by the ith channel,
is composed of
Corresponding to the transmitted signal, the LPF is a low pass filter,
is the frequency of the difference frequency signal,
the phase of the difference frequency signal is A, and the amplitude of the difference frequency signal is A;
the difference frequency signal in the form of the discrete-time signal is:
wherein the content of the first and second substances,
the difference signal being in the form of a discrete-time signal of the ith channel, N s In order to sample the number of points in a discrete time,
for discrete time angular frequency, F s Is a discrete time sampling frequency.
4. The millimeter wave radar gait recognition method based on retrieval task of claim 1, wherein the step 2 comprises:
step 2.1, mixing T x ×R x Difference of discrete time signal form of each channelCarrying out incoherent accumulation on the frequency signals to obtain signals after incoherent accumulation;
step 2.2, performing discrete Fourier transform on the non-coherent accumulated signals along a distance dimension to obtain the time and distance graph;
step 2.3, summing the time and distance graphs according to columns and averaging to obtain a reference vector;
2.4, subtracting the reference vector from each time dimension of the time and distance graph to obtain a time and distance graph after static clutter is filtered;
step 2.5, summing the time after the static noise is filtered and the distance unit where the human body is located in the distance graph, and performing short-time Fourier transform on the vector obtained by summing along a time dimension to obtain a time-frequency graph, wherein the time-frequency graph is as follows:
wherein, X 0 (t, f) is a time-frequency diagram, w (t) is a window function, t is time, f is frequency, m is a sampling point in the slow time dimension, N 0 The total number of distance units occupied by the human body.
5. The millimeter wave radar gait recognition method based on the retrieval task of claim 1, wherein the step 3 comprises:
step 3.1, carrying out nonlinear normalization operation on the time-frequency diagram to obtain a time-frequency diagram of the nonlinear normalization operation, wherein the time-frequency diagram of the nonlinear normalization operation of P persons is total;
and 3.2, taking the time-frequency diagram of at least one part of linear normalization operation in the P human bodies as data in the first data set.
6. The millimeter wave radar gait recognition method based on retrieval task of claim 1, wherein the step 4 comprises:
step 4.1, inputting a reference sample, a positive sample and a negative sample in the training sample set into the gait recognition network model, wherein the reference sample and the positive sample are different samples of the same human body, and the reference sample and the negative sample are different samples of different human bodies;
step 4.2, processing the ternary loss function of the gait recognition network model by using a random gradient descent algorithm to minimize the ternary loss function;
and 4.3, acquiring a trained gait recognition network model through the minimum ternary loss function.
7. The millimeter wave radar gait recognition method based on the retrieval task of claim 6, wherein the ResNet18 includes four sequentially connected residual modules, wherein a feature map output by the previous residual module is input to the next residual module, and a feature map output by each residual module is input to the time-frequency map time dimension feature extraction module, the time-frequency map time dimension feature extraction module correspondingly outputs a plurality of first feature vectors, all the first feature vectors corresponding to all the residual modules are subjected to dimension transformation by using a full connection layer to obtain a plurality of second feature vectors, and all the second feature vectors are subjected to head-to-tail splicing to obtain 1 gait feature vector.
8. The millimeter wave radar gait recognition method based on the retrieval task as claimed in claim 7, wherein the processing method of the time-frequency graph time dimension feature extraction module comprises:
s1.1, dividing a feature map with the dimension of C multiplied by H multiplied by W into N parts along the width direction to obtain N parts of sub-feature maps, wherein C is the number of channels, H is the height of the feature map, and W is the width of the feature map;
s1.2, for each sub-feature diagram
Using global maximum pooling in dimensionality, wherein each sub-feature map becomes a sub-feature vector with the dimensionality of C, and the feature map becomes N third feature vectors with the dimensionality of C;
s1.3, averaging the N third feature vectors with the dimensionality of C to obtain 1 first feature vector with the dimensionality of C.
9. The millimeter wave radar gait recognition method based on retrieval task of claim 1, characterized by further comprising, after step 4:
inputting all samples in a sample library into a trained gait recognition network model to obtain a first step state feature vector, wherein the samples in the sample library are all normal samples;
inputting one sample in query samples into a trained gait recognition network model to obtain a second-step state feature vector, wherein the query samples comprise samples worn normally, dressed in overcoat and worn in a satchel;
sequentially calculating Euclidean distances between the second step state feature vector and the first step state feature vectors of all samples in the sample library;
selecting the sample in the sample library with the minimum Euclidean distance.
10. The millimeter wave radar gait recognition method based on the retrieval task of claim 1, wherein the step 5 comprises:
step 5.1, inputting a time-frequency diagram to be recognized into a trained gait recognition network model to obtain a gait feature vector;
step 5.2, sequentially calculating Euclidean distances between the gait feature vector and gait feature vectors of all samples in a preset sample library;
and 5.3, selecting the sample in the preset sample library with the minimum Euclidean distance as a final identification result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210641204.8A CN115079118A (en) | 2022-06-08 | 2022-06-08 | Millimeter wave radar gait recognition method based on retrieval task |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210641204.8A CN115079118A (en) | 2022-06-08 | 2022-06-08 | Millimeter wave radar gait recognition method based on retrieval task |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115079118A true CN115079118A (en) | 2022-09-20 |
Family
ID=83251277
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210641204.8A Pending CN115079118A (en) | 2022-06-08 | 2022-06-08 | Millimeter wave radar gait recognition method based on retrieval task |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115079118A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116453227A (en) * | 2023-06-19 | 2023-07-18 | 武汉理工大学 | Gait recognition method based on double millimeter wave radar under ship environment |
-
2022
- 2022-06-08 CN CN202210641204.8A patent/CN115079118A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116453227A (en) * | 2023-06-19 | 2023-07-18 | 武汉理工大学 | Gait recognition method based on double millimeter wave radar under ship environment |
| CN116453227B (en) * | 2023-06-19 | 2023-09-19 | 武汉理工大学 | Gait recognition method based on double millimeter wave radar under ship environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Du et al. | Micro-Doppler feature extraction based on time-frequency spectrogram for ground moving targets classification with low-resolution radar | |
| Pegoraro et al. | Real-time people tracking and identification from sparse mm-wave radar point-clouds | |
| Du et al. | Segmented convolutional gated recurrent neural networks for human activity recognition in ultra-wideband radar | |
| CN106814358A (en) | The system and method for many human body target detections in a kind of radar for ultra-broadband wall-through | |
| CN106127110B (en) | A kind of human body fine granularity motion recognition method based on UWB radar and optimal SVM | |
| Kim et al. | Moving target classification in automotive radar systems using convolutional recurrent neural networks | |
| Kim et al. | Human detection based on time-varying signature on range-Doppler diagram using deep neural networks | |
| Molchanov | Radar target classification by micro-Doppler contributions | |
| CN106597411A (en) | Radar signal processing method | |
| CN113391282A (en) | Human body posture recognition method based on radar multi-dimensional feature fusion | |
| CN110647788A (en) | Human daily behavior classification method based on micro-Doppler characteristics | |
| EP2100153B1 (en) | Intelligent biometric radar system | |
| Budillon et al. | SAR tomography based on deep learning | |
| Chen et al. | LFM signal detection and estimation based on deep convolutional neural network | |
| CN115079118A (en) | Millimeter wave radar gait recognition method based on retrieval task | |
| CN113762130A (en) | Millimeter wave radar gesture detection and recognition method | |
| CN106468772A (en) | A kind of multistation radar human body tracing method based on range Doppler measurement | |
| CN116338684A (en) | Human body falling detection method and system based on millimeter wave radar and deep learning | |
| JP2024514618A (en) | DNN-based human face classification | |
| Liang et al. | A posture recognition based fall detection system using a 24GHz CMOS FMCW radar SoC | |
| Xiaolong et al. | A lightweight network model for human activity classifiction based on pre-trained mobilenetv2 | |
| CN114325623A (en) | Method for measuring human body limb movement information based on millimeter wave radar | |
| Li et al. | Gait recognition using spatio-temporal information of 3D point cloud via millimeter wave radar | |
| CN114580473A (en) | Radar-based human behavior identification method and system | |
| Zhao et al. | Interference suppression based gesture recognition method with FMCW radar |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |