CN111259700A

CN111259700A - Method and apparatus for generating gait recognition model

Info

Publication number: CN111259700A
Application number: CN201811463430.1A
Authority: CN
Inventors: 刘武; 梅涛; 程昱昊
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2018-12-03
Filing date: 2018-12-03
Publication date: 2020-06-09
Anticipated expiration: 2038-12-03
Also published as: CN111259700B

Abstract

The embodiment of the application discloses a method and a device for generating a gait recognition model. One embodiment of the method comprises: acquiring a training sample set, wherein the training sample set comprises a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectrums representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing gaits of different users; and training based on the training sample set and a preset loss function to obtain a gait recognition model. This embodiment can improve the accuracy of gait recognition.

Description

Method and apparatus for generating gait recognition model

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a method and a device for generating a gait recognition model.

Background

Gait recognition is a newer biometric recognition technology that has been focused on by more and more researchers in recent years, and is a method for performing identity recognition through the walking posture of people. Gait refers to the way people walk, and is a complex behavioral characteristic. Compared with other biological recognition technologies, gait recognition has the advantages of being non-contact, far-distance and not easy to disguise, and has advantages over image recognition in certain fields (such as the field of intelligent video monitoring).

The related art generally performs gait recognition based on visual signals. For example, features related to gait are extracted directly from captured pictures. However, in practical applications, factors such as ambient light and obstacles may affect the accuracy of gait recognition.

Disclosure of Invention

The embodiment of the application provides a method and a device for generating a gait recognition model.

In a first aspect, an embodiment of the present application provides a method for generating a gait recognition model, the method including: acquiring a training sample set, wherein the training sample set comprises a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectrums representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing gaits of different users; and training based on the training sample set and a preset loss function to obtain a gait recognition model.

In some embodiments, training based on the training sample set and a preset loss function to obtain a gait recognition model includes: the following training steps are performed: for a training sample in a training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained first characteristic information set, the second characteristic information set and the third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, training based on the training sample set and a preset loss function to obtain a gait recognition model further includes: and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the training step.

In some embodiments, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum are selected from a pre-stored set of wireless spectra, a wireless spectrum of the set of wireless spectra generated by: during the walking of a sample user along a preset route, carrying out wireless signal acquisition on a target position of an area where the sample user is located to obtain a wireless signal described by CSI; denoising the acquired wireless signals; and performing time-frequency transformation on the denoised wireless signals to obtain a wireless spectrum representing the gait of the sample user.

In some embodiments, the wireless signals include Wi-Fi signals or millimeter wave signals.

In a second aspect, embodiments of the present application provide a method for identifying gait, the method including: acquiring a wireless spectrum representing the gait of a user to be identified; inputting the wireless spectrum into a gait recognition model generated by adopting the method described by any one implementation mode in the first aspect to obtain the characteristic information of the user to be recognized; matching the characteristic information with pre-stored characteristic information; and generating an identification result of the user to be identified based on the matching result.

In a third aspect, an embodiment of the present application provides a method for generating a gait recognition model, where the gait recognition model includes a first sub-recognition model and a second sub-recognition model, and the method includes: acquiring a first training sample set, wherein the first training sample set comprises a first step state energy graph, a second step state energy graph and a third step state energy graph, the first step state energy graph and the second step state energy graph are gait energy graphs representing gaits of the same user, and the first step state energy graph and the third step state energy graph are gait energy graphs representing gaits of different users; acquiring a second training sample set, wherein the second training sample set comprises a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectrums representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing gaits of different users; training based on a first training sample set and a preset loss function to obtain a first sub-recognition model; training based on a second training sample set and a loss function to obtain a second sub-recognition model; and determining the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In some embodiments, training the first sub-recognition model based on the first training sample set and a preset loss function includes: performing a first training step as follows: for a first training sample in a first training sample set, respectively inputting a first step energy diagram, a second step energy diagram and a third step energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first step energy diagram, second step energy diagram and third step energy diagram; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained first characteristic information set, the second characteristic information set and the third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a first sub-recognition model.

In some embodiments, training based on the first training sample set and a preset loss function to obtain the first sub-recognition model further includes: and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuously executing the first training step.

In some embodiments, training based on the second set of training samples and the loss function results in a second sub-recognition model, comprising: performing a second training step as follows: for a second training sample in a second training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the second training sample into an initial convolutional neural network to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained fourth characteristic information set, the fifth characteristic information set and the sixth characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, training based on the second set of training samples and the loss function to obtain the second sub-recognition model further comprises: and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the second training step.

In some embodiments, the first, second, and third step energy maps are selected from a set of pre-stored gait energy maps, the gait energy maps in the set of gait energy maps being generated by: during the walking of the sample user along the preset route, video acquisition is carried out on the sample user to obtain a walking video of the sample user; extracting a plurality of key frames from the walking video; extracting the human body outline of the sample user from the plurality of key frames to generate a human body silhouette; and synthesizing a gait energy map based on the generated human silhouette.

In a fourth aspect, embodiments of the present application provide a method for identifying gait, the method including: acquiring a gait energy map and a wireless spectrum representing the gait of a user to be identified; inputting the gait energy diagram into a first sub-recognition model of the gait recognition model generated by adopting the method described in any one implementation mode in the third aspect to obtain first characteristic information of the user to be recognized; inputting the wireless spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized; matching the first characteristic information with pre-stored first characteristic information; matching the second characteristic information with pre-stored second characteristic information; and fusing the matching result of the first characteristic information and the matching result of the second characteristic information based on the weight of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model to generate the recognition result of the user to be recognized.

In a fifth aspect, an embodiment of the present application provides a method for generating a gait recognition model, where the gait recognition model includes a first sub-recognition model, a second sub-recognition model and a third sub-recognition model, and the method includes: acquiring a first training sample set, wherein the first training sample set comprises a first step state energy graph, a second step state energy graph and a third step state energy graph, the first step state energy graph and the second step state energy graph are gait energy graphs representing gaits of the same user, and the first step state energy graph and the third step state energy graph are gait energy graphs representing gaits of different users; acquiring a second training sample set, wherein the second training sample set comprises a first Wi-Fi frequency spectrum, a second Wi-Fi frequency spectrum and a third Wi-Fi frequency spectrum, the first Wi-Fi frequency spectrum and the second Wi-Fi frequency spectrum are Wi-Fi frequency spectrums representing gaits of the same user, and the first Wi-Fi frequency spectrum and the third Wi-Fi frequency spectrum are Wi-Fi frequency spectrums representing gaits of different users; acquiring a third training sample set, wherein the third training sample comprises a first millimeter wave frequency spectrum, a second millimeter wave frequency spectrum and a third millimeter wave frequency spectrum, the first millimeter wave frequency spectrum and the second millimeter wave frequency spectrum are millimeter wave frequency spectrums representing gaits of the same user, and the first millimeter wave frequency spectrum and the third millimeter wave frequency spectrum are millimeter wave frequency spectrums representing gaits of different users; training based on a first training sample set and a preset loss function to obtain a first sub-recognition model; training based on a second training sample set and a loss function to obtain a second sub-recognition model; training based on a third training sample set and a loss function to obtain a third sub-recognition model; and determining the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set and the number of the third training samples in the third training sample set.

In a sixth aspect, embodiments of the present application provide a method for identifying gait, the method including: acquiring a gait energy map, a Wi-Fi frequency spectrum and a millimeter wave frequency spectrum which represent the gait of a user to be identified; inputting the gait energy diagram into a first sub-recognition model of the gait recognition model generated by adopting the method described in the fifth aspect to obtain first characteristic information of the user to be recognized; inputting the Wi-Fi frequency spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized; inputting the millimeter wave frequency spectrum into a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized; matching the first characteristic information with pre-stored first characteristic information; matching the second characteristic information with pre-stored second characteristic information; matching the third characteristic information with prestored third characteristic information; and fusing the matching result of the first characteristic information, the matching result of the second characteristic information and the matching result of the third characteristic information based on the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, and generating the recognition result of the user to be recognized.

In a seventh aspect, an embodiment of the present application provides an apparatus for generating a gait recognition model, where the apparatus includes: a sample acquisition unit configured to acquire a training sample set, wherein the training sample set includes a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectra representing gaits of a same user, and the first wireless spectrum and the third wireless spectrum are wireless spectra representing gaits of different users; and the model training unit is configured to train to obtain the gait recognition model based on the training sample set and a preset loss function.

In some embodiments, the model training unit comprises: a training module configured to perform the training steps of: for a training sample in a training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained first characteristic information set, the second characteristic information set and the third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, the model training unit further comprises: and the adjusting module is configured to adjust the parameters of the initial convolutional neural network if the value of the loss function is larger than a preset value, use the adjusted initial convolutional neural network as the initial convolutional neural network, and continue to execute the training step.

In an eighth aspect, an embodiment of the present application provides an apparatus for identifying gait, including: an information acquisition unit configured to acquire a wireless spectrum representing a gait of a user to be identified; the identification unit is configured to input the wireless spectrum into a gait identification model generated by adopting the method described in any one of the implementation manners of the first aspect, and obtain characteristic information of a user to be identified; a matching unit configured to match the feature information with pre-stored feature information; and a result generation unit configured to generate an identification result of the user to be identified based on the matching result.

In a ninth aspect, an embodiment of the present application provides an apparatus for generating a gait recognition model, where the gait recognition model includes a first sub-recognition model and a second sub-recognition model, and the apparatus includes: a first sample acquisition unit configured to acquire a first training sample set, wherein the first training sample set includes a first step energy map, a second step energy map, and a third step energy map, the first step energy map and the second step energy map are gait energy maps representing gaits of the same user, and the first step energy map and the third step energy map are gait energy maps representing gaits of different users; a second sample acquisition unit configured to acquire a second training sample set, wherein the second training sample set includes a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectra representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectra representing gaits of different users; the first model training unit is configured to train based on a first training sample set and a preset loss function to obtain a first sub-recognition model; the second model training unit is configured to train on the basis of a second training sample set and the loss function to obtain a second sub-recognition model; and the weight determination unit is configured to determine the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In some embodiments, the first model training unit comprises: a first training module configured to perform a first training step of: for a first training sample in a first training sample set, respectively inputting a first step energy diagram, a second step energy diagram and a third step energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first step energy diagram, second step energy diagram and third step energy diagram; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained first characteristic information set, the second characteristic information set and the third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a first sub-recognition model.

In some embodiments, the first model training unit further comprises: and the first adjusting module is configured to adjust the parameters of the initial convolutional neural network if the value of the loss function is larger than a preset value, use the adjusted initial convolutional neural network as the initial convolutional neural network, and continue to execute the first training step.

In some embodiments, the second model training unit comprises: a second training module configured to perform a second training step of: for a second training sample in a second training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the second training sample into an initial convolutional neural network to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained fourth characteristic information set, the fifth characteristic information set and the sixth characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, the second model training unit further comprises: and the second adjusting module is configured to adjust the parameters of the initial convolutional neural network if the value of the loss function is larger than a preset value, use the adjusted initial convolutional neural network as the initial convolutional neural network, and continue to execute the second training step.

In a tenth aspect, an embodiment of the present application provides an apparatus for identifying gait, including: an information acquisition unit configured to acquire a gait energy map and a wireless spectrum characterizing a gait of a user to be identified; a first identification unit, configured to input a gait energy map into a first sub-identification model of a gait identification model generated by the method described in any one of the third aspects, so as to obtain first feature information of a user to be identified; the second identification unit is configured to input the wireless spectrum into a second sub-identification model of the gait identification model to obtain second characteristic information of the user to be identified; a first matching unit configured to match the first feature information with pre-stored first feature information; a second matching unit configured to match the second feature information with pre-stored second feature information; and the result generation unit is configured to fuse the matching result of the first characteristic information and the matching result of the second characteristic information based on the weight of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generate the recognition result of the user to be recognized.

In an eleventh aspect, an embodiment of the present application provides an apparatus for generating a gait recognition model, where the gait recognition model includes a first sub-recognition model, a second sub-recognition model, and a third sub-recognition model, and the apparatus includes: a first sample acquisition unit configured to acquire a first training sample set, wherein the first training sample set includes a first step energy map, a second step energy map, and a third step energy map, the first step energy map and the second step energy map are gait energy maps representing gaits of the same user, and the first step energy map and the third step energy map are gait energy maps representing gaits of different users; a second sample acquisition unit configured to acquire a second training sample set, wherein the second training sample set includes a first Wi-Fi spectrum, a second Wi-Fi spectrum, and a third Wi-Fi spectrum, the first Wi-Fi spectrum and the second Wi-Fi spectrum are Wi-Fi spectra representing gaits of the same user, and the first Wi-Fi spectrum and the third Wi-Fi spectrum are Wi-Fi spectra representing gaits of different users; the third sample acquisition unit is configured to acquire a third training sample set, wherein the third training sample set comprises a first millimeter wave spectrum, a second millimeter wave spectrum and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectra representing gaits of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectra representing gaits of different users; the first model training unit is configured to train based on a first training sample set and a preset loss function to obtain a first sub-recognition model; the second model training unit is configured to train on the basis of a second training sample set and the loss function to obtain a second sub-recognition model; a third model training unit configured to obtain a third sub-recognition model based on a third training sample set and the loss function training; and the weight determination unit is configured to determine the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set and the number of the third training samples in the third training sample set.

In a twelfth aspect, an embodiment of the present application provides an apparatus for identifying gait, including: the information acquisition unit is configured to acquire a gait energy map, a Wi-Fi frequency spectrum and a millimeter wave frequency spectrum which represent the gait of the user to be identified; a first identification unit, configured to input a gait energy map into a first sub-identification model of the gait identification model generated by the method described in the fifth aspect to obtain first feature information of the user to be identified; the second identification unit is configured to input the Wi-Fi frequency spectrum into a second sub-identification model of the gait identification model to obtain second characteristic information of the user to be identified; the third identification unit is configured to input the millimeter wave frequency spectrum into a third sub-identification model of the gait identification model to obtain third characteristic information of the user to be identified; a first matching unit configured to match the first feature information with pre-stored first feature information; a second matching unit configured to match the second feature information with pre-stored second feature information; a third matching unit configured to match the third feature information with pre-stored third feature information; and the result generation unit is configured to fuse the matching result of the first characteristic information, the matching result of the second characteristic information and the matching result of the third characteristic information based on the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, and generate the recognition result of the user to be recognized.

In a thirteenth aspect, an embodiment of the present application provides an electronic device, including: one or more processors; a storage device having one or more programs stored thereon; when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any one of the implementations of the first aspect to the sixth aspect.

In a fourteenth aspect, the present application provides a computer-readable medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the method as described in any implementation manner of the first aspect to the sixth aspect.

According to the method and the device for generating the gait recognition model, the gait recognition accuracy can be improved by obtaining the training sample set of the training samples generated by the wireless spectrum representing the gait of the same user and the wireless spectrum representing the gait of different users, and then training the training sample set and the preset loss function to obtain the gait recognition model.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present application may be applied;

FIG. 2 is a flow diagram of one embodiment of a method for generating a gait recognition model according to the application;

FIG. 3 is a flow chart of one embodiment of a method for identifying gait according to the application;

fig. 4 is a flow diagram of another embodiment of a method for generating a gait recognition model according to the application;

FIG. 5 is a flow chart of another embodiment of a method for identifying gait according to the application;

FIG. 6 is a flow chart of yet another embodiment of a method for generating a gait recognition model according to the application;

FIG. 7 is a flow chart of yet another embodiment of a method for identifying gait according to the application;

figures 8 and 9 are schematic diagrams of an application scenario of a method for generating a gait recognition model according to the application;

FIG. 10 is a schematic diagram illustrating the structure of one embodiment of an apparatus for generating a gait recognition model according to the application;

FIG. 11 is a schematic diagram illustrating the structure of one embodiment of an apparatus for identifying gait according to the application;

fig. 12 is a schematic structural diagram of another embodiment of an apparatus for generating a gait recognition model according to the application;

FIG. 13 is a schematic diagram of another embodiment of an apparatus for identifying gait according to the application;

fig. 14 is a schematic structural diagram of a further embodiment of an apparatus for generating a gait recognition model according to the application;

FIG. 15 is a schematic structural diagram of yet another embodiment of an apparatus for identifying gait according to the application;

FIG. 16 is a block diagram of a computer system suitable for use in implementing the electronic device of an embodiment of the present application.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of the method for generating a gait recognition model, the method for recognizing gait, the apparatus for generating a gait recognition model or the apparatus for recognizing gait of the present application can be applied.

As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and

servers

105, 106. The network 104 is used to provide a medium for communication links between the

terminal devices

101, 102, 103 and the

servers

105, 106. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the

servers

105, 106 via the network 104 to receive or send messages or the like. Various communication client applications, such as a gait recognition application, a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like, may be installed on the

terminal devices

101, 102, 103.

The

terminal apparatuses

101, 102, and 103 may be hardware or software. When the

terminal devices

101, 102, 103 are hardware, they may be various electronic devices that support gait recognition, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like. When the

terminal apparatuses

101, 102, 103 are software, they can be installed in the electronic apparatuses listed above. It may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.

The server 105 may be a server that provides various services, such as a data server that stores training samples. The data server may store a first set of training samples. The first training samples may include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum. The first wireless spectrum and the second wireless spectrum are wireless spectrums representing the gait of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing the gait of different users.

The server 105 may also store a second set of training samples. The second training sample may include a first step energy map, a second step energy map, and a third step energy map. The first step state energy diagram and the second step state energy diagram are gait energy diagrams representing gaits of the same user, and the first step state energy diagram and the third step state energy diagram are gait energy diagrams representing gaits of different users.

The server 106 may be a server providing various services, such as a background server providing support for gait recognition-like applications on the

terminal devices

101, 102, 103. The background server may train the model to be trained by using the training sample set stored in the data server 105 to obtain the gait recognition model. The background server can also input the information (such as wireless spectrum) representing the gait of the user to be identified, which is submitted by the terminal equipment, into the gait identification model to obtain the characteristic information of the user to be identified, and generate an identification result by utilizing the characteristic information output by the model and the pre-stored characteristic information.

It should be noted that the method for generating a gait recognition model or the method for recognizing gait provided in the embodiments of the present application is generally performed by the server 106, and accordingly, the device for generating a gait recognition model or the device for recognizing gait is generally disposed in the server 106.

The

servers

105 and 106 may be hardware or software. When the

servers

105 and 106 are hardware, they may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When the server is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.

It should be noted that the training sample set may also be stored locally by the server 106, and the training sample set may be directly obtained by the server 106. At this point, the exemplary system architecture 100 may not include the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any suitable number of terminal devices, networks, and servers, as desired for implementation.

With continued reference to fig. 2, a flow 200 of one embodiment of a method for generating a gait recognition model according to the application is shown. The method for generating the gait recognition model can comprise the following steps:

step 201, a training sample set is obtained.

In this embodiment, the executing agent (e.g., server 106 of fig. 1) of the method for generating a gait recognition model may obtain the set of training samples from a local or remote location. Wherein the training samples may include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum. The first wireless spectrum and the second wireless spectrum are wireless spectra characterizing gait of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectra characterizing gait of different users. As an example, in a certain training sample, the first wireless spectrum and the second wireless spectrum may represent gait information of "zhangsan" of the sample user, and the third wireless spectrum may represent gait information of "liquan" of the sample user.

In some optional implementations of this embodiment, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum may be selected from a pre-stored set of wireless spectra (e.g., a set of wireless spectra pre-stored on a server).

Corresponding to this implementation, the wireless spectrum in the set of wireless spectrums may be generated by:

firstly, during the period that a sample user walks along a preset route, wireless signal acquisition is carried out on a target position of an area where the sample user is located, and a wireless signal described by Channel State Information (CSI) is obtained. Here, the target position may be a specific position located on the wireless signal propagation path. The CSI describes the signal change of a wireless signal from transmission to reception, and the CSI description information is changed by human body characteristics and movement.

Alternatively, the Wireless signals may include Wi-Fi (Wireless-Fidelity) signals or millimeter wave signals. In one example, a wireless router may be used to collect Wi-Fi signals and then use CSI to describe the collected Wi-Fi signals. In another example, millimeter wave signals may be collected using a millimeter wave transmit/receive device, and then the collected millimeter wave signals may be described using CSI.

Then, denoising processing is carried out on the acquired wireless signals. As an example, a PCA (principal component Analysis) technique or other suitable technique may be used to remove noise in the wireless signal described by the CSI, thereby improving the accuracy of the wireless signal.

And finally, performing time-frequency transformation on the denoised wireless signals to obtain a wireless spectrum representing the gait of the sample user. As an example, a Short Time fourier transform (Short Time fourier transform) may be performed on the denoised wireless signal to convert the wireless signal from a Time domain to a frequency domain, thereby obtaining a wireless spectrum for characterizing the gait of the sample user.

In this implementation, the execution subject of the wireless spectrum generation step may be the same as or different from the execution subject of the method for generating the gait recognition model. If so, the executing subject of the wireless-spectrum generating step may store the set of wireless spectrum locally after generating the set of wireless spectrum. If not, the performing agent of the wireless-spectrum generating step may transmit the set of wireless spectrum to the performing agent of the method for generating the gait recognition model after generating the set of wireless spectrum.

And 202, training based on the training sample set and a preset loss function to obtain a gait recognition model.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for generating a gait recognition model may train an initial convolutional neural network by using a machine learning method using a training sample set and a preset loss function to obtain the gait recognition model. Here, the initial Convolutional Neural Network may use various existing Convolutional Neural Network (CNN) structures. Such as DenseNet, GoogleNet, VGGNet, ResNet, etc.

In some optional implementations of this embodiment, step 202 may specifically include:

in a first step, the following training steps are performed:

firstly, for each training sample in a training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the training sample to an initial convolutional neural network, so as to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum. Here, the first feature information, the second feature information, and the third feature information may be information representing human gait, and may be represented by a vector or a matrix.

Then, based on the obtained first feature information set, second feature information set, and third feature information set, it is determined whether a value of a preset loss function (for example, a value of a loss function may be obtained by inputting each of the first feature information, second feature information, and third feature information into a preset loss function) is less than or equal to a preset numerical value.

And if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

And secondly, if the value of the loss function is larger than a preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the training step.

In the process of training the initial convolutional neural network, the partial derivative of the loss function to the weight of each neuron can be calculated layer by layer to form the gradient of the loss function to the weight vector, so as to modify the weight of the initial convolutional neural network. The learning of the gait recognition model is completed in the weight modifying process. When the value of the loss function reaches a desired value (e.g., less than 0.05), the training of the gait recognition model is complete.

It should be noted that the objective of training the initial convolutional neural network is to make the difference of the feature information extracted from the wireless spectrum representing the gait of the same user as small as possible, and at the same time, to make the difference of the feature information extracted from the wireless spectrum representing the gait of different users as large as possible. Here, the difference of the feature information may be characterized by using the similarity (e.g., euclidean distance, cosine similarity, etc.) of the feature information. Here, a difference in characteristic information extracted from a wireless spectrum representing the gait of the same user may be referred to as a first difference, and a difference in characteristic information extracted from a wireless spectrum representing the gait of a different user may be referred to as a second difference. The Loss function may be a function for characterizing a degree of difference between the second difference and the first difference, such as a triple Loss function (triple Loss). As an example, a Triplet Loss may be defined as:

where N is a natural number greater than 1, i is the number of the training sample, i ═ 1,2, …, N },

a first radio spectrum, a second radio spectrum, and a third radio spectrum for an ith training sample,

is a first wireless spectrum

The corresponding first characteristic information is transmitted to the mobile terminal,

is the second wireless spectrum

The corresponding second characteristic information is then transmitted to the second device,

is the third wireless spectrum

The corresponding third characteristic information, α, is a constant.

In the embodiment, a triple consisting of two wireless frequency spectrums representing the gaits of the same user and two wireless frequency spectrums representing the gaits of different users is used as a training sample, and the trained gait recognition model is not influenced by ambient illumination, obstacles and the like, so that the accuracy of gait recognition is improved.

In the method for generating the gait recognition model according to the above embodiment of the present application, the training sample set including the training samples generated by the wireless spectrum representing the gait of the same user and the wireless spectrum representing the gait of different users is obtained, and then the gait recognition model is obtained by training based on the training sample set and the preset loss function, so that the accuracy of gait recognition can be improved.

With further reference to fig. 3, a flow 300 of one embodiment of a method for identifying gait according to the application is shown. The method for recognizing gait can comprise the following steps:

step 301, a wireless spectrum representing the gait of the user to be identified is obtained.

In this embodiment, an executing subject of the method for identifying gait (e.g., the server 106 of fig. 1) may acquire a wireless spectrum for characterizing the gait of the user to be identified.

In some optional implementation manners of this embodiment, step 301 may specifically include:

firstly, during the walking of a user to be identified along a preset route, wireless signal acquisition is carried out on a target position of an area where the user to be identified is located, and a wireless signal described by CSI is obtained. Here, the wireless signal may include a Wi-Fi signal or a millimeter wave signal.

Secondly, denoising the acquired wireless signals.

And finally, performing time-frequency transformation on the denoised wireless signal to obtain a wireless spectrum representing the gait of the user to be identified.

It should be noted that the radio spectrum for characterizing the gait of the user to be identified may also be acquired by a terminal device or a server different from the executing entity, and then transmitted to the executing entity.

Step 302, inputting the wireless spectrum into the gait recognition model to obtain the characteristic information of the user to be recognized.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for recognizing gait may input the wireless spectrum acquired in step 301 into a trained gait recognition model, so as to obtain feature information of the user to be recognized. The gait recognition model can be generated by the method described in the corresponding embodiment of fig. 2. The characteristic information can be information representing human gait and can be represented by a vector or a matrix.

And step 303, matching the characteristic information with pre-stored characteristic information.

In the present embodiment, an executing body (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the above-described feature information with pre-stored feature information. The pre-stored characteristic information may be pre-extracted from a wireless spectrum representing the gait of the user. The execution subject may determine a similarity between the feature information and pre-stored feature information (e.g., may be determined by using an euclidean distance, a cosine similarity, etc.). If the similarity is greater than or equal to a preset value, it can be determined that the characteristic information is matched with the pre-stored characteristic information. If the similarity is smaller than a preset value, it can be determined that the characteristic information is not matched with the pre-stored characteristic information.

And 304, generating an identification result of the user to be identified based on the matching result.

In the present embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may generate the recognition result of the user to be recognized based on the matching result. For example, if the characteristic information is matched with pre-stored characteristic information, an identification result indicating that the identity authentication is passed may be generated; if the characteristic information does not match the pre-stored characteristic information, an identification result indicating that the identity authentication fails can be generated.

According to the method for recognizing the gait provided by the embodiment of the application, the acquired wireless spectrum for representing the gait of the user to be recognized is input into the trained gait recognition model, then the characteristic information output by the model is matched with the pre-stored characteristic information, and finally the recognition result of the user to be recognized is generated based on the matching result, so that the accuracy of gait recognition can be improved.

With further reference to fig. 4, a flow 400 of another embodiment of a method for generating a gait recognition model according to the application is shown. The method for generating the gait recognition model can comprise the following steps:

step 401, a first training sample set is obtained.

In this embodiment, the executing subject (e.g., server 106 of fig. 1) of the method for generating a gait recognition model may obtain the first set of training samples locally or remotely. The first training sample may include a first step energy map, a second step energy map, and a third step energy map. The first step energy map and the second step energy map are gait energy maps representing gaits of the same user, and the first step energy map and the third step energy map are gait energy maps representing gaits of different users.

Gait Energy Image (GEI) is a common feature in Gait detection, the extraction method is simple, and features such as Gait speed, Gait morphology and the like can be well expressed. As an example, a gait energy map may be defined as:

where x and y are pixel coordinates, M is the total number of key frames, t is the number of key frames, {1,2, …, M }, G (x, y) represents the final gait energy map, which is a two-dimensional image of x and y, and I (x, y, t) is the pixel value (e.g., grayscale value) of the pixel with coordinates (x, y) in the key frame with number t.

In some optional implementations of the present embodiment, the first step energy map, the second step energy map, and the third step energy map may be selected from a set of pre-stored gait energy maps.

Corresponding to this implementation, the gait energy map in the set of gait energy maps can be generated by:

firstly, during the walking of a sample user along a preset route, video acquisition is carried out on the sample user to obtain a walking video of the sample user. For example, a camera may be used to capture walking video of a sample user.

Then, a plurality of key frames are extracted from the walking video. As an example, the required key frames may be extracted from the walking video in 10 frames per second.

Then, the body contour of the sample user is extracted from the plurality of key frames, and a body silhouette (also referred to as a gait silhouette) is generated. For example, a moving object (i.e., a sample user) may be extracted from the keyframes using a human segmentation method.

Finally, a gait energy map is synthesized based on the generated gait silhouettes.

In this implementation, the execution subject of the gait energy map generation step may be the same as or different from the execution subject of the method for generating the gait recognition model. If so, the executing entity of the gait energy map generating step may store the set of gait energy maps locally after generating the set of gait energy maps. If not, the executive body of the gait energy map generation step may send the set of gait energy maps after generation of the set of gait energy maps to the executive body of the method for generating a gait recognition model.

Step 402, a second set of training samples is obtained.

In this embodiment, the executing subject of the method for generating a gait recognition model (e.g. the server 106 of fig. 1) may obtain the second set of training samples from locally or remotely. Wherein the second training samples may include the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum. The first wireless spectrum and the second wireless spectrum are wireless spectra characterizing gait of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectra characterizing gait of different users.

In some optional implementations of this embodiment, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum may be selected from a pre-stored set of wireless spectra. The generation step of the radio spectrum in the radio spectrum set may refer to the description about the generation step of the radio spectrum in the embodiment corresponding to fig. 2, and is not described herein again.

In some optional implementations of this embodiment, the wireless signal includes a Wi-Fi signal or a millimeter wave signal.

And 403, training to obtain a first sub-recognition model based on the first training sample set and a preset loss function.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for generating a gait recognition model may train an initial convolutional neural network by using a machine learning method using a first training sample set and a preset loss function, so as to obtain a first sub-recognition model. Here, the initial convolutional neural network may use various existing convolutional neural network structures.

In some optional implementations of this embodiment, step 403 may specifically include:

in a first step, a first training step is performed as follows:

firstly, for each first training sample in a first training sample set, respectively inputting a first step energy diagram, a second step energy diagram and a third step energy diagram of the first training sample into an initial convolutional neural network, and obtaining first feature information, second feature information and third feature information corresponding to the input first step energy diagram, second step energy diagram and third step energy diagram. Here, the first feature information, the second feature information, and the third feature information may be information representing human gait, and may be represented by a vector or a matrix.

And if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a first sub-recognition model.

And 404, training based on the second training sample set and the loss function to obtain a second sub-recognition model.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for generating a gait recognition model may train the initial convolutional neural network by using a machine learning method using the second training sample set and a preset loss function, so as to obtain a second sub-recognition model. Here, the initial convolutional neural network may use various existing convolutional neural network structures.

In some optional implementations of this embodiment, step 404 may specifically include:

in a first step, a second training step is performed as follows:

firstly, for each second training sample in a second training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the second training sample to an initial convolutional neural network, so as to obtain fourth feature information, fifth feature information and sixth feature information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum. Here, the fourth feature information, the fifth feature information, and the sixth feature information may be information representing human gait, and may be represented by a vector or a matrix.

Then, based on the obtained fourth, fifth, and sixth feature information sets, it is determined whether a value of a preset loss function (for example, a value of a loss function may be obtained by inputting each of the fourth, fifth, and sixth feature information into the preset loss function) is less than or equal to a preset numerical value.

And if the value of the loss function is smaller than or equal to the preset value, determining the initial convolutional neural network as a second sub-recognition model.

And secondly, if the value of the loss function is larger than a preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the second training step.

In this embodiment, the same initial convolutional neural network and loss function training may be used to obtain the first sub-recognition model and the second sub-recognition model, or different initial convolutional neural networks and loss functions training may be used to obtain the first sub-recognition model and the second sub-recognition model, which is not specifically limited in this application.

Step 405, determining weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for generating the gait recognition model may determine the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set. As an example, if the number of first training samples in the first training sample set is 3000 and the number of second training samples in the second training sample set is 1000, it may be determined that the weight ratio of the model output of the first sub-recognition model to the model output of the second sub-recognition model in the output of the gait recognition model is 3: 1.

As can be seen from fig. 4, compared with the embodiment corresponding to fig. 2, the flow 400 of the method for generating a gait recognition model in the present embodiment embodies the step of training the first sub-recognition model of the gait recognition model using the gait energy map. Therefore, the scheme described in the embodiment can make up for application scenarios (for example, different walking routes, different wearing, and the like) in which the gait recognition model trained by using the wireless spectrum has a low recognition rate, so as to further improve the accuracy of gait recognition.

With further reference to fig. 5, a flow 500 of another embodiment of a method for identifying gait according to the application is shown. The method for recognizing gait can comprise the following steps:

step 501, acquiring a gait energy map and a wireless spectrum for representing the gait of a user to be identified.

In this embodiment, an executing body (e.g., the server 106 of fig. 1) of the method for recognizing gait can acquire a gait energy map and a wireless spectrum for characterizing the gait of the user to be recognized.

In some optional implementations of this embodiment, step 501 may specifically include:

firstly, during the period that a user to be identified walks along a preset route, video acquisition is carried out on the user to be identified, and a walking video of the user to be identified is obtained. For example, a camera may be used to capture walking video of a user to be identified.

Then, the human body contour of the user to be identified is extracted from the plurality of key frames, and a human body silhouette (also called a gait silhouette) is generated. For example, a moving object (i.e., a user to be identified) may be extracted from the key frame using a human body segmentation method.

In some optional implementations of this embodiment, step 501 may further include:

Secondly, denoising the acquired wireless signals.

It should be noted that the gait energy map and the wireless spectrum for characterizing the gait of the user to be identified may also be acquired by a terminal device or a server different from the executing subject and then transmitted to the executing subject.

Step 502, inputting the gait energy map into a first sub-recognition model of the gait recognition model to obtain first characteristic information of the user to be recognized.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for recognizing gait may input the gait energy map obtained in step 501 into a first sub-recognition model of the trained gait recognition model, so as to obtain first feature information of the user to be recognized. The gait recognition model can be generated by the method described in the embodiment corresponding to fig. 4. The first characteristic information may be information characterizing human gait, and may be represented by a vector or a matrix.

Step 503, inputting the wireless spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for recognizing gait may input the wireless spectrum acquired in step 501 into a second sub-recognition model of the trained gait recognition model, so as to obtain second feature information of the user to be recognized. The second characteristic information may be information representing human gait, and may be represented by a vector or a matrix.

And step 504, matching the first characteristic information with pre-stored first characteristic information.

In the present embodiment, an executing subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the above-described first feature information with pre-stored first feature information. The pre-stored first characteristic information may be pre-extracted from a gait energy map representing a gait of the user. Here, matching the first feature information with pre-stored first feature information may determine a similarity between the first feature information and the pre-stored first feature information (e.g., may be determined using an euclidean distance, a cosine similarity, or the like).

And 505, matching the second characteristic information with pre-stored second characteristic information.

In the present embodiment, the performing subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the above-mentioned second feature information with pre-stored second feature information. Wherein, the pre-stored second characteristic information may be pre-extracted from a wireless spectrum characterizing the gait of the user. Here, the similarity between the second feature information and the pre-stored second feature information may be determined (for example, the similarity may be determined using euclidean distance, cosine, or the like).

And 506, fusing the matching result of the first characteristic information and the matching result of the second characteristic information based on the weight of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generating the recognition result of the user to be recognized.

In this embodiment, an executing entity (for example, the server 106 in fig. 1) of the method for recognizing gait may fuse the matching result of the first feature information and the matching result of the second feature information to generate a recognition result of the user to be recognized. As an example, the weight ratio of the model output of the first sub-recognition model to the model output of the second sub-recognition model in the output of the gait recognition model is 3:1, the similarity of the first feature information output by the first sub-recognition model to the pre-stored first feature information is 90%, the similarity of the second feature information output by the second sub-recognition model to the pre-stored second feature information is 75%, and the recognition result of the user to be recognized may be 90% × 0.75+ 75% × 0.25 — 86.25%. If the similarity threshold is 85%, the identification result may indicate that the identity authentication of the user to be identified passes.

As can be seen from fig. 5, compared with the embodiment corresponding to fig. 3, the flow 500 of the method for identifying gait in this embodiment embodies the steps of acquiring the gait energy map and the wireless spectrum of the user to be identified, and fusing the identification results thereof to generate the identification result. Therefore, the scheme described in the embodiment can make up for application scenarios (for example, different walking routes, different wearing, and the like) in which the gait recognition model trained by using the wireless spectrum has a low recognition rate, so as to further improve the accuracy of gait recognition.

With further reference to fig. 6, a flow 600 of yet another embodiment of a method for generating a gait recognition model according to the application is shown. The method for generating the gait recognition model can comprise the following steps:

step 601, a first training sample set is obtained.

Step 602, a second set of training samples is obtained.

In this embodiment, the executing subject of the method for generating a gait recognition model (e.g. the server 106 of fig. 1) may obtain the second set of training samples from locally or remotely. Wherein the second training samples may include a first Wi-Fi spectrum, a second Wi-Fi spectrum, and a third Wi-Fi spectrum. The first Wi-Fi frequency spectrum and the second Wi-Fi frequency spectrum are Wi-Fi frequency spectrums representing gaits of the same user, and the first Wi-Fi frequency spectrum and the third Wi-Fi frequency spectrum are Wi-Fi frequency spectrums representing gaits of different users.

Step 603, a third training sample set is obtained.

In this embodiment, the executing subject of the method for generating a gait recognition model (e.g. the server 106 of fig. 1) may obtain the third set of training samples from locally or remotely. Wherein the third training samples may include the first millimeter wave spectrum, the second millimeter wave spectrum, and the third millimeter wave spectrum. The first millimeter wave frequency spectrum and the second millimeter wave frequency spectrum are millimeter wave frequency spectrums representing the gait of the same user, and the first millimeter wave frequency spectrum and the third millimeter wave frequency spectrum are millimeter wave frequency spectrums representing the gait of different users.

Step 604, training based on the first training sample set and a preset loss function to obtain a first sub-recognition model.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for generating a gait recognition model may train an initial convolutional neural network by using a machine learning method using a first training sample set and a preset loss function to obtain the gait recognition model. Here, the initial convolutional neural network may use various existing convolutional neural network structures.

For the specific training step of the first sub-recognition model, reference may be made to the description of the training step of the first sub-recognition model in the embodiment corresponding to fig. 4, which is not repeated herein.

Step 605, training based on the second training sample set and the loss function to obtain a second sub-recognition model.

And 606, training based on the third training sample set and the loss function to obtain a third sub-recognition model.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for generating a gait recognition model may train an initial convolutional neural network by using a machine learning method using a third training sample set and a preset loss function, so as to obtain a third sub-recognition model. Here, the initial convolutional neural network may use various existing convolutional neural network structures.

The specific training steps of the second sub-recognition model and the third sub-recognition model may refer to the description about the training step of the second sub-recognition model in the embodiment corresponding to fig. 4, and are not described herein again.

Step 607, determining weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set and the number of the third training samples in the third training sample set.

In this embodiment, an executive (e.g., the server 106 in fig. 1) of the method for generating a gait recognition model may determine weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set and the number of the third training samples in the third training sample set. As an example, if the number of first training samples in the first training sample set is 3000, the number of second training samples in the second training sample set is 1000, and the number of third training samples in the third training sample set is 1000, it may be determined that the weight ratio of the model outputs of the first sub-recognition model, the second sub-recognition model, and the third sub-recognition model in the output of the gait recognition model is 3:1: 1.

As can be seen from fig. 6, compared with the embodiment corresponding to fig. 2, the flow 600 of the method for generating a gait recognition model in the present embodiment embodies the steps of training the first sub-recognition model, the second sub-recognition model and the third sub-recognition model of the gait recognition model using the gait energy map, the Wi-Fi spectrum and the millimeter wave spectrum. Therefore, the scheme described in the embodiment can make up for application scenarios (for example, different walking routes, different wearing, and the like) in which the gait recognition model trained by using a single wireless spectrum is not high in recognition rate, so as to further improve the accuracy of gait recognition.

With further reference to fig. 7, a flow 700 of yet another embodiment of a method for identifying gait according to the application is shown. The method for recognizing gait can comprise the following steps:

step 701, acquiring a gait energy map, a Wi-Fi frequency spectrum and a millimeter wave frequency spectrum for representing the gait of the user to be identified.

In this embodiment, an executing subject of the method for recognizing gait (e.g., the server 106 of fig. 1) may acquire a gait energy map Wi-Fi spectrum and a millimeter wave spectrum for characterizing the gait of the user to be recognized.

Step 702, inputting the gait energy map into a first sub-recognition model of the gait recognition model to obtain first characteristic information of the user to be recognized.

In this embodiment, an executing subject (e.g., the server 106 in fig. 1) of the method for recognizing gait may input the acquired gait energy map into a first sub-recognition model of the trained gait recognition model, so as to obtain first feature information of the user to be recognized. The gait recognition model can be generated by the method described in the corresponding embodiment of fig. 6. The first characteristic information may be information characterizing human gait, and may be represented by a vector or a matrix.

And 703, inputting the Wi-Fi frequency spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized.

In this embodiment, an executing subject (e.g., the server 106 in fig. 1) of the method for recognizing gait may input the acquired Wi-Fi spectrum into a second sub-recognition model of the trained gait recognition model, so as to obtain second feature information of the user to be recognized. The second characteristic information may be information representing human gait, and may be represented by a vector or a matrix.

Step 704, inputting the millimeter wave frequency spectrum into a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized.

In this embodiment, an executing entity (e.g., the server 106 in fig. 1) of the method for recognizing gait may input the acquired millimeter wave spectrum into a third sub-recognition model of the trained gait recognition model, so as to obtain third feature information of the user to be recognized. The third characteristic information may be information representing human gait, and may be represented by a vector or a matrix.

Step 705, matching the first characteristic information with pre-stored first characteristic information.

And step 706, matching the second characteristic information with pre-stored second characteristic information.

In the present embodiment, the performing subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the above-mentioned second feature information with pre-stored second feature information. Wherein, the pre-stored second characteristic information can be pre-extracted from a Wi-Fi frequency spectrum representing the gait of the user. Here, the similarity between the second feature information and the pre-stored second feature information may be determined (for example, the similarity may be determined using euclidean distance, cosine, or the like).

And step 707, matching the third characteristic information with the pre-stored third characteristic information.

In the present embodiment, the performing subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the above-mentioned third feature information with pre-stored third feature information. The pre-stored third characteristic information may be pre-extracted from a millimeter wave spectrum representing the gait of the user. Here, the third feature information may be matched with the prestored third feature information, and the similarity between the third feature information and the prestored third feature information may be determined (for example, the similarity may be determined by using an euclidean distance, a cosine similarity, or the like).

Step 708, based on the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, the matching result of the first feature information, the matching result of the second feature information and the matching result of the third feature information are fused to generate the recognition result of the user to be recognized.

In this embodiment, an executing entity (for example, the server 106 in fig. 1) of the method for recognizing gait may fuse the matching result of the first feature information, the matching result of the second feature information, and the matching result of the third feature information to generate a recognition result of the user to be recognized. As an example, the weight ratio of the model outputs of the first sub-recognition model, the second sub-recognition model and the third sub-recognition model in the output of the gait recognition model is 3:1:1, the similarity of the first characteristic information output by the first sub-recognition model to the pre-stored first characteristic information is 90%, the similarity of the second characteristic information output by the second sub-recognition model to the pre-stored second characteristic information is 75%, and the similarity of the third characteristic information output by the third sub-recognition model to the pre-stored third characteristic information is 87%, so that the recognition result of the user to be recognized may be 90% × 0.6+ 75% × 0.2+ 87% × 0.2 — 86.4%. If the similarity threshold is 85%, the identification result may indicate that the identity authentication of the user to be identified passes.

As can be seen from fig. 7, compared with the embodiment corresponding to fig. 3, the flow 700 of the method for identifying gait in this embodiment embodies the steps of acquiring the gait energy map, the Wi-Fi spectrum and the millimeter wave spectrum of the user to be identified, and fusing the identification results thereof to generate the identification result. Therefore, the scheme described in the embodiment can make up for application scenarios (for example, different walking routes, different wearing, and the like) in which the gait recognition model trained by using a single wireless spectrum is not high in recognition rate, so as to further improve the accuracy of gait recognition.

With continuing reference to fig. 8 and 9, fig. 8 and 9 are schematic diagrams of one application scenario of a method for generating a gait recognition model according to the application.

As shown in fig. 8, first, three preset routes L are provided in the room shown by the solid line box₁～L₃The system comprises three cameras 802-804, a wireless router 801, a radio frequency receiving end 805 and a radio frequency transmitting end 806. Then, 50 experimenters 807 are invited to respectively follow the preset route L₁～L₃The walking video is collected by the cameras 802-804, the walking video of the experimenter 807 is collected by the wireless router 801, Wi-Fi signals of each experimenter during walking are collected by the millimeter wave radio frequency receiving end 805, millimeter wave signals transmitted by the millimeter wave radio frequency transmitting end 806 of each experimenter during walking are collected, and finally, for example, 9831 walking videos, 3277 Wi-Fi signals and 3277 millimeter wave signals can be collected, wherein each walking video and each wireless signal comprise a plurality of Gait cycles (a goal Cycle refers to a walking process that one foot strides out from the heel to the heel of the next time during walking). Then, each walking video is processed to obtain a gait energy map (shown as 901 in fig. 9), each Wi-Fi signal is processed to obtain a Wi-Fi spectrum (shown as 902 in fig. 9), and each millimeter wave signal is processed to obtain a millimeter wave spectrum (shown as 903 in fig. 9). Then, a gait energy map set, a Wi-Fi frequency spectrum set and a millimeter wave frequency spectrum set are used for respectively generating first training samplesThe training sample set comprises a present set, a second training sample set and a third training sample set. Then, the first sub-recognition model, the second sub-recognition model and the third sub-recognition model are trained respectively, and the weight ratio (e.g. 3:1:1) of the model outputs of the first sub-recognition model, the second sub-recognition model and the third sub-recognition model in the output of the gait recognition model is determined. Finally, a gait recognition model, i.e. a combination of the first sub-recognition model, the second sub-recognition model and the third sub-recognition model, which determine the weight ratio, is obtained.

With further reference to fig. 10, as an implementation of the method shown in fig. 2, the present application provides an embodiment of an apparatus for generating a gait recognition model, which corresponds to the embodiment of the method shown in fig. 2, and which is particularly applicable in a server.

As shown in fig. 10, the apparatus 1000 for generating a gait recognition model of the present embodiment may include a sample acquisition unit 1001 and a model training unit 1002. Wherein the sample acquiring unit 1001 is configured to acquire a training sample set, wherein the training samples include a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectrums representing gaits of a same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing gaits of different users; and the model training unit 1002 is configured to train to obtain a gait recognition model based on the training sample set and a preset loss function.

In this embodiment, for specific implementation of the sample obtaining unit 1001 and the model training unit 1002 of the apparatus 1000 for generating a gait recognition model, reference may be made to the related description about step 201 to step 202 in the embodiment corresponding to fig. 2, and details are not repeated here.

In some optional implementations of this embodiment, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum may be selected from a pre-stored set of wireless spectra. The wireless spectrum in the set of wireless spectra is generated by: during the walking of a sample user along a preset route, carrying out wireless signal acquisition on a target position of an area where the sample user is located to obtain a wireless signal described by CSI; denoising the acquired wireless signals; and performing time-frequency transformation on the denoised wireless signals to obtain a wireless spectrum representing the gait of the sample user.

In some optional implementations of this embodiment, the wireless signal may comprise a Wi-Fi signal or a millimeter wave signal.

In some optional implementations of this embodiment, the model training unit 1002 may include a training module. Wherein the training module may be configured to perform the following training steps: for a training sample in a training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained first characteristic information set, the second characteristic information set and the third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some optional implementations of this embodiment, the model training unit 1002 may further include an adjusting module. Wherein the adjustment module may be configured to: and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuously executing the training step.

The device for generating the gait recognition model according to the above embodiment of the present application obtains the training sample set including the training samples generated by the wireless spectrum representing the gait of the same user and the wireless spectrum representing the gait of different users, and then trains to obtain the gait recognition model based on the training sample set and the preset loss function, thereby improving the accuracy of gait recognition.

With further reference to fig. 11, as an implementation of the method shown in fig. 3, the present application provides an embodiment of an apparatus for identifying gait, which corresponds to the embodiment of the method shown in fig. 3, and which can be applied in a server in particular.

As shown in fig. 11, the apparatus 1100 for recognizing gait of the present embodiment may include an information acquisition unit 1101, a recognition unit 1102, a matching unit 1103, and a result generation unit 1104. Wherein the information acquisition unit 1101 is configured to acquire a wireless spectrum representing a gait of the user to be identified; the identifying unit 1102 is configured to input the wireless spectrum to a gait recognition model, and obtain characteristic information of a user to be identified; the matching unit 1103 is configured to match the feature information with pre-stored feature information; and the result generation unit 1104 is configured to generate an identification result of the user to be identified based on the matching result.

In this embodiment, for specific implementation of the information acquiring unit 1101, the identifying unit 1102, the matching unit 1103 and the result generating unit 1104 of the device 1100 for identifying gait, reference may be made to the related description of step 301 to step 304 in the embodiment corresponding to fig. 3, and details are not described here again.

According to the device for identifying the gait provided by the embodiment of the application, the acquired wireless spectrum for representing the gait of the user to be identified is input into the trained gait identification model, then the characteristic information output by the model is matched with the pre-stored characteristic information, and finally the identification result of the user to be identified is generated based on the matching result, so that the accuracy of gait identification can be improved.

With further reference to fig. 12, as an implementation of the method shown in fig. 4, the present application provides another embodiment of an apparatus for generating a gait recognition model, which corresponds to the embodiment of the method shown in fig. 4, and which may be applied in a server in particular.

As shown in fig. 12, the apparatus 1200 for generating a gait recognition model of the present embodiment may include a first sample acquisition unit 1201, a second sample acquisition unit 1202, a first model training unit 1203, a second model training unit 1204, and a weight determination unit 1205. Wherein the first sample obtaining unit 1201 is configured to obtain a first set of training samples, wherein the first training samples comprise a first step energy map, a second step energy map and a third step energy map, the first step energy map and the second step energy map are gait energy maps representing gaits of the same user, the first step energy map and the third step energy map are gait energy maps representing gaits of different users; the second sample obtaining unit 1202 is configured to obtain a second training sample set, where the second training sample includes a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectra that characterize gait of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectra that characterize gait of different users; the first model training unit 1203 is configured to train to obtain a first sub-recognition model based on the first training sample set and a preset loss function; the second model training unit 1204 is configured to derive a second sub-recognition model based on a second set of training samples and the loss function training; and the weight determination unit 1205 is configured to determine the weight of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In this embodiment, for specific implementations of the first sample acquiring unit 1201, the second sample acquiring unit 1202, the first model training unit 1203, the second model training unit 1204, and the weight determining unit 1205 of the apparatus 1200 for generating a gait recognition model, reference may be made to the related descriptions of steps 401 to 405 in the embodiment corresponding to fig. 4, and details are not repeated here.

In some alternative implementations of the present embodiment, the first, second, and third step energy maps are selected from a set of pre-stored gait energy maps. The gait energy map in the set of gait energy maps can be generated by: during the walking of the sample user along the preset route, video acquisition is carried out on the sample user to obtain a walking video of the sample user; extracting a plurality of key frames from the walking video; extracting the human body outline of the sample user from the plurality of key frames to generate a human body silhouette; and synthesizing a gait energy map based on the generated human silhouette.

In some optional implementations of this embodiment, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum are selected from a pre-stored set of wireless spectra. The wireless spectrum in the set of wireless spectra may be generated by: during the walking of a sample user along a preset route, carrying out wireless signal acquisition on a target position of an area where the sample user is located to obtain a wireless signal described by CSI; denoising the acquired wireless signals; and performing time-frequency transformation on the denoised wireless signals to obtain a wireless spectrum representing the gait of the sample user.

In some optional implementations of the present embodiment, the first model training unit 1203 may include a first training module. Wherein the first training module may be configured to perform a first training step of: for a first training sample in a first training sample set, respectively inputting a first step energy diagram, a second step energy diagram and a third step energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first step energy diagram, second step energy diagram and third step energy diagram; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained first characteristic information set, the second characteristic information set and the third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a first sub-recognition model.

In some optional implementations of this embodiment, the first model training unit 1203 may further include a first adjusting module. Wherein the first adjusting module is configured to: and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuously executing the first training step.

In some optional implementations of this embodiment, the second model training unit 1204 may include a second training module. Wherein the second training module is configured to perform a second training step of: for a second training sample in a second training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the second training sample into an initial convolutional neural network to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum; determining whether the value of the loss function is less than or equal to a preset value or not based on the obtained fourth characteristic information set, the fifth characteristic information set and the sixth characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some optional implementations of this embodiment, the second model training unit 1204 may further include a second adjusting module. Wherein the second adjustment module is configured to: and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the second training step.

The device for generating the gait recognition model provided by the above embodiment of the application can make up for application scenarios (for example, different walking routes, different wearing, etc.) with low recognition rate of the gait recognition model trained by using the wireless spectrum, thereby further improving the accuracy rate of gait recognition.

With further reference to fig. 13, as an implementation of the method shown in fig. 5, the present application provides another embodiment of an apparatus for identifying gait, which corresponds to the embodiment of the method shown in fig. 5, and which can be applied in a server in particular.

As shown in fig. 13, the apparatus 1300 for recognizing gait of the present embodiment may include an information acquisition unit 1301, a first recognition unit 1302, a second recognition unit 1303, a first matching unit 1304, a second matching unit 1305, and a result generation unit 1306. Wherein, the information obtaining unit 1301 is configured to obtain a gait energy map and a radio spectrum characterizing the gait of the user to be identified; the first recognition unit 1302 is configured to input a gait energy map into a first sub-recognition model of a gait recognition model to obtain first feature information of a user to be recognized; the second identifying unit 1303 is configured to input the wireless spectrum into a second sub-identifying model of the gait identifying model to obtain second feature information of the user to be identified; the first matching unit 1304 is configured to match the first feature information with pre-stored first feature information; the second matching unit 1305 is configured to match the second feature information with pre-stored second feature information; and the result generation unit 1306 is configured to fuse the matching result of the first feature information and the matching result of the second feature information based on a weight of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generate a recognition result of the user to be recognized.

In this embodiment, for specific implementation of the information obtaining unit 1301, the first identifying unit 1302, the second identifying unit 1303, the first matching unit 1304, the second matching unit 1305, and the result generating unit 1306 of the apparatus 1300 for identifying gait, reference may be made to the related description of steps 501 to 506 in the embodiment corresponding to fig. 5, and details are not repeated here.

The device for recognizing gait provided by the above embodiment of the application can make up for application scenarios (for example, different walking routes, different wearing, etc.) with low recognition rate of the gait recognition model trained by using the wireless spectrum, thereby further improving the accuracy rate of gait recognition.

With further reference to fig. 14, as an implementation of the method shown in fig. 6, the present application provides a further embodiment of an apparatus for generating a gait recognition model, which corresponds to the embodiment of the method shown in fig. 6, and which may be applied in a server in particular.

As shown in fig. 14, the apparatus 1400 for generating a gait recognition model of the present embodiment may include a first sample acquisition unit 1401, a second sample acquisition unit 1402, a third sample acquisition unit 1403, a first model training unit 1404, a second model training unit 1405, a third model training unit 1406, and a weight determination unit 1407. Wherein the first sample acquiring unit 1401 is configured to acquire a first set of training samples, wherein the first training samples comprise a first step energy map, a second step energy map and a third step energy map, the first step energy map and the second step energy map are gait energy maps representing gaits of the same user, the first step energy map and the third step energy map are gait energy maps representing gaits of different users; the second sample obtaining unit 1402 is configured to obtain a second set of training samples, wherein the second training samples include a first Wi-Fi spectrum, a second Wi-Fi spectrum, and a third Wi-Fi spectrum, the first Wi-Fi spectrum and the second Wi-Fi spectrum are Wi-Fi spectrums characterizing gait of the same user, and the first Wi-Fi spectrum and the third Wi-Fi spectrum are Wi-Fi spectrums characterizing gait of different users; the third sample obtaining unit 1403 is configured to obtain a third training sample set, where the third training sample includes a first millimeter wave spectrum, a second millimeter wave spectrum, and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectra that represent gaits of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectra that represent gaits of different users; the first model training unit 1404 is configured to train to obtain a first sub-recognition model based on the first training sample set and a preset loss function; the second model training unit 1405 is configured to derive a second sub-recognition model based on the second set of training samples and the loss function training; the third model training unit 1406 is configured to train to obtain a third sub-recognition model based on the third training sample set and the loss function; the weight determining unit 1407 is configured to determine weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model, and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set, and the number of the third training samples in the third training sample set.

In this embodiment, for specific implementations of the first sample acquiring unit 1401, the second sample acquiring unit 1402, the third sample acquiring unit 1403, the first model training unit 1404, the second model training unit 1405, the third model training unit 1406, and the weight determining unit 1407 of the apparatus 1400 for generating a gait recognition model, reference may be made to the related descriptions of steps 601 to 607 in the embodiment corresponding to fig. 6, and no further description is given here.

The device for generating the gait recognition model according to the above embodiment of the application can make up for application scenarios (for example, different walking routes, different wearing, etc.) in which the recognition rate of the gait recognition model trained by using a single radio spectrum is not high, thereby further improving the accuracy of gait recognition.

With further reference to fig. 15, as an implementation of the method shown in fig. 7, the present application provides yet another embodiment of an apparatus for identifying gait, which corresponds to the method embodiment shown in fig. 7, and which can be applied in a server in particular.

As shown in fig. 15, the apparatus 1500 for recognizing gait of the present embodiment may include an information acquisition unit 1501, a first recognition unit 1502, a second recognition unit 1503, a third recognition unit 1504, a first matching unit 1505, a second matching unit 1506, a third matching unit 1507, and a result generation unit 1508. Wherein the information acquisition unit 1501 is configured to acquire a gait energy map, a Wi-Fi spectrum and a millimeter wave spectrum characterizing a gait of the user to be identified; the first recognition unit 1502 is configured to input the gait energy map into a first sub-recognition model of the gait recognition model to obtain first feature information of the user to be recognized; the second identification unit 1503 is configured to input the Wi-Fi spectrum to a second sub-identification model of the gait identification model to obtain second characteristic information of the user to be identified; the third identification unit 1504 is configured to input the millimeter wave frequency spectrum into a third sub-identification model of the gait identification model to obtain third feature information of the user to be identified; the first matching unit 1505 is configured to match the first feature information with pre-stored first feature information; the second matching unit 1506 is configured to match the second feature information with pre-stored second feature information; the third matching unit 1507 is configured to match the third feature information with pre-stored third feature information; the result generating unit 1508 is configured to fuse the matching result of the first feature information, the matching result of the second feature information, and the matching result of the third feature information based on the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model, and the model output of the third sub-recognition model in the output of the gait recognition model, and generate a recognition result of the user to be recognized.

In this embodiment, specific implementations of the information acquisition unit 1501, the first identification unit 1502, the second identification unit 1503, the third identification unit 1504, the first matching unit 1505, the second matching unit 1506, the third matching unit 1507, and the result generation unit 1508 of the apparatus 1500 for identifying gait may refer to the related descriptions of step 701 to step 708 in the embodiment corresponding to fig. 7, and are not described herein again.

The device for recognizing gait provided by the above embodiment of the application can make up for application scenarios (for example, different walking routes, different wearing, etc.) with low recognition rate of the gait recognition model trained by using a single wireless spectrum, thereby further improving the accuracy rate of gait recognition.

Referring now to FIG. 16, a block diagram of a computer system 1600 suitable for use in implementing an electronic device (e.g., server 106 of FIG. 1) of an embodiment of the present application is shown. The electronic device shown in fig. 16 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

As shown in fig. 16, the computer system 1600 includes one or more Central Processing Units (CPUs) 1601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)1602 or a program loaded from a storage portion 1608 into a Random Access Memory (RAM) 1603. In the RAM 1603, various programs and data necessary for the operation of the system 1600 are also stored. The CPU 1601, ROM 1602, and RAM 1603 are connected to each other via a bus 1604. An input/output (I/O) interface 1605 is also connected to the bus 1604.

The following components are connected to the I/O interface 1605: an input portion 1606 including a keyboard, a mouse, and the like; an output portion 1607 including a display device such as an Organic Light Emitting Diode (OLED) display, a Liquid Crystal Display (LCD), and a speaker; a storage portion 1608 including a hard disk and the like; and a communication section 1609 including a network interface card such as a LAN card, a modem, or the like. The communication section 1609 performs communication processing via a network such as the internet. The driver 1610 is also connected to the I/O interface 1605 as needed. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1610 as necessary, so that a computer program read out therefrom is mounted in the storage portion 1608 as necessary.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 1609, and/or installed from the removable media 1611. The above-described functions defined in the method of the present application are executed when the computer program is executed by the Central Processing Unit (CPU) 1601.

It should be noted that the computer readable medium described herein can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present application may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a sample acquisition unit and a model training unit. Where the names of these units do not in some cases constitute a limitation on the units themselves, for example, a sample acquisition unit may also be described as a "unit that acquires a set of training samples".

As another aspect, the present application also provides a computer-readable medium, which may be contained in the apparatus described in the above embodiments; or may be present separately and not assembled into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: acquiring a training sample set, wherein the training sample set comprises a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectrums representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing gaits of different users; and training based on the training sample set and a preset loss function to obtain a gait recognition model.

The above description is only a preferred embodiment of the application and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention herein disclosed is not limited to the particular combination of features described above, but also encompasses other arrangements formed by any combination of the above features or their equivalents without departing from the spirit of the invention. For example, the above features may be replaced with (but not limited to) features having similar functions disclosed in the present application.

Claims

1. A method for generating a gait recognition model, comprising:

acquiring a training sample set, wherein the training sample set comprises a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectrums representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing gaits of different users;

and training based on the training sample set and a preset loss function to obtain the gait recognition model.

2. The method according to claim 1, wherein the training based on the training sample set and a preset loss function to obtain the gait recognition model comprises:

the following training steps are performed:

for the training samples in the training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the training samples to an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum;

determining whether the value of the loss function is smaller than or equal to a preset value or not based on the obtained first characteristic information set, the second characteristic information set and the third characteristic information set;

and if the value of the loss function is smaller than or equal to the preset value, determining the initial convolutional neural network as the gait recognition model.

3. The method of claim 2, wherein the training based on the training sample set and a preset loss function to obtain the gait recognition model further comprises:

and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the training step.

4. The method of claim 1, wherein the first, second, and third wireless spectrums are selected from a set of prestored wireless spectrums, a wireless spectrum in the set of wireless spectrums being generated by:

during the walking of a sample user along a preset route, carrying out wireless signal acquisition on a target position of an area where the sample user is located to obtain a wireless signal described by Channel State Information (CSI);

denoising the acquired wireless signals;

and performing time-frequency transformation on the denoised wireless signals to obtain a wireless spectrum representing the gait of the sample user.

5. The method of claim 4, wherein the wireless signals comprise wireless fidelity (Wi-Fi) signals or millimeter wave signals.

6. A method for identifying gait, comprising:

acquiring a wireless spectrum representing the gait of a user to be identified;

inputting the wireless spectrum into a gait recognition model generated by the method of any one of claims 1 to 5 to obtain characteristic information of the user to be recognized;

matching the characteristic information with pre-stored characteristic information;

and generating an identification result of the user to be identified based on the matching result.

7. A method for generating a gait recognition model comprising a first sub-recognition model and a second sub-recognition model, the method comprising:

acquiring a first training sample set, wherein the first training sample set comprises a first step state energy graph, a second step state energy graph and a third step state energy graph, the first step state energy graph and the second step state energy graph are gait energy graphs representing gaits of the same user, and the first step state energy graph and the third step state energy graph are gait energy graphs representing gaits of different users;

acquiring a second training sample set, wherein the second training sample set comprises a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectrums representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums representing gaits of different users;

training based on the first training sample set and a preset loss function to obtain the first sub-recognition model;

training based on the second training sample set and the loss function to obtain the second sub-recognition model;

and determining the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of first training samples in the first training sample set and the number of second training samples in the second training sample set.

8. The method of claim 7, wherein the training the first sub-recognition model based on the first set of training samples and a preset loss function comprises:

performing a first training step as follows:

for a first training sample in the first training sample set, respectively inputting a first step energy diagram, a second step energy diagram and a third step energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first step energy diagram, second step energy diagram and third step energy diagram;

and if the value of the loss function is smaller than or equal to the preset value, determining the initial convolutional neural network as the first sub-recognition model.

9. The method of claim 8, wherein the training based on the first set of training samples and a preset loss function to obtain the first sub-recognition model further comprises:

and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the first training step.

10. The method of claim 7, wherein said training the second sub-recognition model based on the second set of training samples and the loss function comprises:

performing a second training step as follows:

for a second training sample in the second training sample set, respectively inputting a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the second training sample to an initial convolutional neural network, so as to obtain fourth feature information, fifth feature information and sixth feature information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum;

determining whether the value of the loss function is smaller than or equal to a preset value or not based on the obtained fourth characteristic information set, the fifth characteristic information set and the sixth characteristic information set;

11. The method of claim 10, wherein the training based on the second set of training samples and the loss function to derive the second sub-recognition model further comprises:

and if the value of the loss function is larger than the preset value, adjusting the parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the second training step.

12. The method of claim 7, wherein the first, second, and third step energy maps are selected from a set of pre-stored gait energy maps, the gait energy maps of the set of gait energy maps being generated by:

during the walking of the sample user along the preset route, video acquisition is carried out on the sample user to obtain a walking video of the sample user;

extracting a plurality of key frames from the walking video;

extracting the human body outline of the sample user from the plurality of key frames to generate a human body silhouette;

and synthesizing a gait energy map based on the generated human silhouette.

13. The method of claim 7, wherein the first, second, and third wireless spectrums are selected from a set of prestored wireless spectrums, a wireless spectrum in the set of wireless spectrums being generated by:

during the walking of a sample user along a preset route, carrying out wireless signal acquisition on a target position of an area where the sample user is located to obtain a wireless signal described by CSI;

denoising the acquired wireless signals;

14. The method of claim 13, wherein the wireless signals comprise Wi-Fi signals or millimeter wave signals.

15. A method for identifying gait, comprising:

acquiring a gait energy map and a wireless spectrum representing the gait of a user to be identified;

inputting the gait energy map into a first sub-recognition model of a gait recognition model generated by the method of any one of claims 7 to 14 to obtain first characteristic information of the user to be recognized;

inputting the wireless spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized;

matching the first characteristic information with pre-stored first characteristic information;

matching the second characteristic information with pre-stored second characteristic information;

and fusing the matching result of the first characteristic information and the matching result of the second characteristic information based on the weight of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generating the recognition result of the user to be recognized.

16. A method for generating a gait recognition model comprising a first sub-recognition model, a second sub-recognition model and a third sub-recognition model, the method comprising:

acquiring a second training sample set, wherein the second training sample set comprises a first Wi-Fi frequency spectrum, a second Wi-Fi frequency spectrum and a third Wi-Fi frequency spectrum, the first Wi-Fi frequency spectrum and the second Wi-Fi frequency spectrum are Wi-Fi frequency spectrums representing gaits of the same user, and the first Wi-Fi frequency spectrum and the third Wi-Fi frequency spectrum are Wi-Fi frequency spectrums representing gaits of different users;

acquiring a third training sample set, wherein the third training sample comprises a first millimeter wave frequency spectrum, a second millimeter wave frequency spectrum and a third millimeter wave frequency spectrum, the first millimeter wave frequency spectrum and the second millimeter wave frequency spectrum are millimeter wave frequency spectrums representing gaits of the same user, and the first millimeter wave frequency spectrum and the third millimeter wave frequency spectrum are millimeter wave frequency spectrums representing gaits of different users;

training based on the third training sample set and the loss function to obtain the third sub-recognition model;

and determining the weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of first training samples in the first training sample set, the number of second training samples in the second training sample set and the number of third training samples in the third training sample set.

17. A method for identifying gait, comprising:

acquiring a gait energy map, a Wi-Fi frequency spectrum and a millimeter wave frequency spectrum which represent the gait of a user to be identified;

inputting the gait energy map into a first sub-recognition model of a gait recognition model generated by the method of claim 16 to obtain first characteristic information of the user to be recognized;

inputting the Wi-Fi frequency spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized;

inputting the millimeter wave frequency spectrum into a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized;

matching the third characteristic information with prestored third characteristic information;

and fusing the matching result of the first characteristic information, the matching result of the second characteristic information and the matching result of the third characteristic information based on the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, and generating the recognition result of the user to be recognized.

18. An apparatus for generating a gait recognition model, comprising:

a sample acquisition unit configured to acquire a training sample set, wherein the training sample set includes a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectra representing gaits of a same user, and the first wireless spectrum and the third wireless spectrum are wireless spectra representing gaits of different users;

a model training unit configured to train to obtain the gait recognition model based on the training sample set and a preset loss function.

19. The apparatus of claim 18, wherein the model training unit comprises:

a training module configured to perform the training steps of:

20. The apparatus of claim 19, wherein the model training unit further comprises:

and the adjusting module is configured to adjust parameters of the initial convolutional neural network if the value of the loss function is larger than the preset value, use the adjusted initial convolutional neural network as the initial convolutional neural network, and continue to execute the training step.

21. The apparatus of claim 18, wherein the first, second, and third wireless spectrums are selected from a set of prestored wireless spectrums, a wireless spectrum in the set of wireless spectrums being generated by:

denoising the acquired wireless signals;

22. The apparatus of claim 21, wherein the wireless signals comprise Wi-Fi signals or millimeter wave signals.

23. An apparatus for identifying gait, comprising:

an information acquisition unit configured to acquire a wireless spectrum representing a gait of a user to be identified;

an identification unit configured to input the wireless spectrum into a gait recognition model generated by the method of any one of claims 1 to 5, and obtain characteristic information of the user to be identified;

a matching unit configured to match the feature information with pre-stored feature information;

a result generation unit configured to generate an identification result of the user to be identified based on a matching result.

24. An apparatus for generating a gait recognition model comprising a first sub-recognition model and a second sub-recognition model, the apparatus comprising:

a first sample acquisition unit configured to acquire a first training sample set, wherein the first training sample set includes a first step energy map, a second step energy map, and a third step energy map, the first step energy map and the second step energy map are gait energy maps representing gaits of the same user, and the first step energy map and the third step energy map are gait energy maps representing gaits of different users;

a second sample acquisition unit configured to acquire a second training sample set, wherein the second training sample set includes a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum are wireless spectra representing gaits of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectra representing gaits of different users;

a first model training unit configured to train to obtain the first sub-recognition model based on the first training sample set and a preset loss function;

a second model training unit configured to train the second sub-recognition model based on the second training sample set and the loss function;

a weight determination unit configured to determine weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model according to the number of first training samples in the first training sample set and the number of second training samples in the second training sample set.

25. The apparatus of claim 24, wherein the first model training unit comprises:

a first training module configured to perform a first training step of:

26. The apparatus of claim 25, wherein the first model training unit further comprises:

and the first adjusting module is configured to adjust parameters of the initial convolutional neural network if the value of the loss function is larger than the preset value, use the adjusted initial convolutional neural network as the initial convolutional neural network, and continue to execute the first training step.

27. The apparatus of claim 24, wherein the second model training unit comprises:

a second training module configured to perform a second training step of:

28. The apparatus of claim 27, wherein the second model training unit further comprises:

and the second adjusting module is configured to adjust the parameters of the initial convolutional neural network if the value of the loss function is greater than the preset value, use the adjusted initial convolutional neural network as the initial convolutional neural network, and continue to execute the second training step.

29. The apparatus of claim 24, wherein the first, second, and third step energy maps are selected from a set of pre-stored gait energy maps, the gait energy maps of the set of gait energy maps being generated by:

extracting a plurality of key frames from the walking video;

and synthesizing a gait energy map based on the generated human silhouette.

30. The apparatus of claim 24, wherein the first, second, and third wireless spectrums are selected from a set of pre-stored wireless spectrums, the wireless spectrums of the set of wireless spectrums generated by:

denoising the acquired wireless signals;

31. The apparatus of claim 30, wherein the wireless signals comprise Wi-Fi signals or millimeter wave signals.

32. An apparatus for identifying gait, comprising:

an information acquisition unit configured to acquire a gait energy map and a wireless spectrum characterizing a gait of a user to be identified;

a first identification unit configured to input the gait energy map into a first sub-identification model of a gait identification model generated by the method of any one of claims 7 to 14 to obtain first characteristic information of the user to be identified;

a second identification unit configured to input the wireless spectrum to a second sub-identification model of the gait identification model to obtain second characteristic information of the user to be identified;

a first matching unit configured to match the first feature information with pre-stored first feature information;

a second matching unit configured to match the second feature information with pre-stored second feature information;

and the result generation unit is configured to fuse the matching result of the first characteristic information and the matching result of the second characteristic information based on the weight of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generate the recognition result of the user to be recognized.

33. An apparatus for generating a gait recognition model comprising a first sub-recognition model, a second sub-recognition model and a third sub-recognition model, the apparatus comprising:

a second sample acquisition unit configured to acquire a second training sample set, wherein the second training sample set includes a first Wi-Fi spectrum, a second Wi-Fi spectrum, and a third Wi-Fi spectrum, the first Wi-Fi spectrum and the second Wi-Fi spectrum are Wi-Fi spectra representing gaits of the same user, and the first Wi-Fi spectrum and the third Wi-Fi spectrum are Wi-Fi spectra representing gaits of different users;

the third sample acquisition unit is configured to acquire a third training sample set, wherein the third training sample set comprises a first millimeter wave spectrum, a second millimeter wave spectrum and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectra representing gaits of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectra representing gaits of different users;

a third model training unit configured to train to obtain the third sub-recognition model based on the third training sample set and the loss function;

a weight determination unit configured to determine weights of a model output of the first sub-recognition model, a model output of the second sub-recognition model, and a model output of the third sub-recognition model in an output of the gait recognition model according to a number of first training samples in the first training sample set, a number of second training samples in the second training sample set, and a number of third training samples in the third training sample set.

34. An apparatus for identifying gait, comprising:

the information acquisition unit is configured to acquire a gait energy map, a Wi-Fi frequency spectrum and a millimeter wave frequency spectrum which represent the gait of the user to be identified;

a first identification unit configured to input the gait energy map into a first sub-identification model of a gait identification model generated by the method of claim 16 to obtain first characteristic information of the user to be identified;

a second identification unit configured to input the Wi-Fi frequency spectrum into a second sub-identification model of the gait identification model to obtain second characteristic information of the user to be identified;

a third identification unit, configured to input the millimeter wave frequency spectrum into a third sub-identification model of the gait identification model to obtain third feature information of the user to be identified;

a third matching unit configured to match the third feature information with pre-stored third feature information;

and the result generation unit is configured to fuse the matching result of the first characteristic information, the matching result of the second characteristic information and the matching result of the third characteristic information based on the weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, and generate the recognition result of the user to be recognized.

35. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-17.

36. A computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the method of any one of claims 1-17.