CN111259700B

CN111259700B - Method and apparatus for generating gait recognition model

Info

Publication number: CN111259700B
Application number: CN201811463430.1A
Authority: CN
Inventors: 刘武; 梅涛; 程昱昊
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Jingdong Shangke Information Technology Co Ltd
Priority date: 2018-12-03
Filing date: 2018-12-03
Publication date: 2024-04-09
Anticipated expiration: 2038-12-03
Also published as: CN111259700A

Abstract

The embodiment of the application discloses a method and a device for generating a gait recognition model. One embodiment of the method comprises the following steps: acquiring a training sample set, wherein the training sample comprises a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum, the first wireless frequency spectrum and the second wireless frequency spectrum are wireless frequency spectrums for representing gait of the same user, and the first wireless frequency spectrum and the third wireless frequency spectrum are wireless frequency spectrums for representing gait of different users; and training based on the training sample set and a preset loss function to obtain a gait recognition model. This embodiment can improve the accuracy of gait recognition.

Description

Method and apparatus for generating gait recognition model

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a method and a device for generating a gait recognition model.

Background

Gait recognition is a newer biometric technology of interest to more and more researchers in recent years, and is a method for performing identity recognition through the gesture of walking by people. Gait refers to a mode of walking of people, and is a complex behavior feature. Gait recognition has the advantages of non-contact, long distance and less camouflage compared to other biometric technologies, and is more advantageous than image recognition in some areas (e.g., intelligent video surveillance).

The related art generally performs gait recognition based on visual signals. For example, features related to gait are extracted directly from captured pictures. However, in practical applications, factors such as illumination of the environment and obstacles may affect the accuracy of gait recognition.

Disclosure of Invention

The embodiment of the application provides a method and a device for generating a gait recognition model.

In a first aspect, embodiments of the present application provide a method for generating a gait recognition model, the method comprising: acquiring a training sample set, wherein the training sample comprises a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum, the first wireless frequency spectrum and the second wireless frequency spectrum are wireless frequency spectrums for representing gait of the same user, and the first wireless frequency spectrum and the third wireless frequency spectrum are wireless frequency spectrums for representing gait of different users; and training based on the training sample set and a preset loss function to obtain a gait recognition model.

In some embodiments, training to obtain a gait recognition model based on the training sample set and a preset loss function includes: the following training steps are performed: for training samples in the training sample set, respectively inputting a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum of the training samples into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless frequency spectrum, second wireless frequency spectrum and third wireless frequency spectrum; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained first characteristic information set, second characteristic information set and third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, training to obtain a gait recognition model based on the training sample set and a preset loss function further comprises: if the value of the loss function is larger than the preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the training step.

In some embodiments, the first, second, and third wireless spectrum are selected from a pre-stored set of wireless spectrums, the wireless spectrums in the set of wireless spectrums being generated by the steps of: during the period that the sample user walks along the preset route, acquiring a wireless signal from a target position of an area where the sample user is located, and acquiring a wireless signal described by adopting the CSI; denoising the acquired wireless signals; and performing time-frequency conversion on the denoised wireless signals to obtain a wireless spectrum representing gait of the sample user.

In some embodiments, the wireless signals include Wi-Fi signals or millimeter wave signals.

In a second aspect, embodiments of the present application provide a method for recognizing gait, the method comprising: acquiring a wireless spectrum representing gait of a user to be identified; inputting the wireless spectrum into a gait recognition model generated by adopting the method described by any implementation manner of the first aspect to obtain the characteristic information of the user to be recognized; matching the characteristic information with pre-stored characteristic information; and generating a recognition result of the user to be recognized based on the matching result.

In a third aspect, embodiments of the present application provide a method for generating a gait recognition model, the gait recognition model including a first sub-recognition model and a second sub-recognition model, the method comprising: acquiring a first training sample set, wherein the first training sample comprises a first gait energy diagram, a second gait energy diagram and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram are gait energy diagrams for representing the gait of the same user, and the first gait energy diagram and the third gait energy diagram are gait energy diagrams for representing the gait of different users; acquiring a second training sample set, wherein the second training sample set comprises a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum, the first wireless frequency spectrum and the second wireless frequency spectrum are wireless frequency spectrums for representing the gait of the same user, and the first wireless frequency spectrum and the third wireless frequency spectrum are wireless frequency spectrums for representing the gait of different users; training to obtain a first sub-recognition model based on a first training sample set and a preset loss function; training based on the second training sample set and the loss function to obtain a second sub-recognition model; and determining weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In some embodiments, training to obtain the first sub-recognition model based on the first training sample set and a preset loss function includes: the following first training step is performed: for a first training sample in a first training sample set, respectively inputting a first gait energy diagram, a second gait energy diagram and a third gait energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first gait energy diagram, second gait energy diagram and third gait energy diagram; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained first characteristic information set, second characteristic information set and third characteristic information set; and if the value of the loss function is smaller than or equal to a preset numerical value, determining the initial convolutional neural network as a first sub-recognition model.

In some embodiments, training to obtain the first sub-recognition model based on the first training sample set and a preset loss function further includes: if the value of the loss function is larger than a preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the first training step.

In some embodiments, training to obtain a second sub-recognition model based on the second set of training samples and the loss function includes: the following second training step is performed: for a second training sample in the second training sample set, respectively inputting a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum of the second training sample into an initial convolutional neural network to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless frequency spectrum, second wireless frequency spectrum and third wireless frequency spectrum; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained fourth characteristic information set, fifth characteristic information set and sixth characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, training to obtain a second sub-recognition model based on the second set of training samples and the loss function further comprises: if the value of the loss function is larger than the preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the second training step.

In some embodiments, the first, second, and third gait energy patterns are selected from a pre-stored set of gait energy patterns, the gait energy patterns in the set of gait energy patterns being generated by: during the walking period of the sample user along the preset route, video acquisition is carried out on the sample user, so that a walking video of the sample user is obtained; extracting a plurality of key frames from the walking video; extracting human body contours of sample users from a plurality of key frames to generate human body silhouettes; and synthesizing a gait energy diagram based on the generated human body silhouette.

In a fourth aspect, embodiments of the present application provide a method for recognizing gait, the method comprising: acquiring a gait energy diagram and a wireless frequency spectrum for representing the gait of a user to be identified; inputting the gait energy diagram into a first sub-recognition model of a gait recognition model generated by adopting the method described by any implementation manner of the third aspect to obtain first characteristic information of a user to be recognized; inputting the wireless frequency spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized; matching the first characteristic information with pre-stored first characteristic information; matching the second characteristic information with pre-stored second characteristic information; and fusing the matching result of the first characteristic information and the matching result of the second characteristic information based on the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, so as to generate a recognition result of the user to be recognized.

In a fifth aspect, embodiments of the present application provide a method for generating a gait recognition model, the gait recognition model including a first sub-recognition model, a second sub-recognition model and a third sub-recognition model, the method comprising: acquiring a first training sample set, wherein the first training sample comprises a first gait energy diagram, a second gait energy diagram and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram are gait energy diagrams for representing the gait of the same user, and the first gait energy diagram and the third gait energy diagram are gait energy diagrams for representing the gait of different users; acquiring a second training sample set, wherein the second training sample set comprises a first Wi-Fi frequency spectrum, a second Wi-Fi frequency spectrum and a third Wi-Fi frequency spectrum, the first Wi-Fi frequency spectrum and the second Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing gait of the same user, and the first Wi-Fi frequency spectrum and the third Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing gait of different users; acquiring a third training sample set, wherein the third training sample set comprises a first millimeter wave spectrum, a second millimeter wave spectrum and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectrums for representing gait of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectrums for representing gait of different users; training to obtain a first sub-recognition model based on a first training sample set and a preset loss function; training based on the second training sample set and the loss function to obtain a second sub-recognition model; training based on the third training sample set and the loss function to obtain a third sub-recognition model; and determining weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set and the number of the third training samples in the third training sample set.

In a sixth aspect, embodiments of the present application provide a method for recognizing gait, the method comprising: acquiring a gait energy diagram, wi-Fi frequency spectrum and millimeter wave frequency spectrum which characterize gait of a user to be identified; inputting the gait energy diagram into a first sub-recognition model of the gait recognition model generated by the method described in the fifth aspect to obtain first characteristic information of a user to be recognized; inputting Wi-Fi frequency spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized; inputting the millimeter wave frequency spectrum into a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized; matching the first characteristic information with pre-stored first characteristic information; matching the second characteristic information with pre-stored second characteristic information; matching the third characteristic information with pre-stored third characteristic information; and fusing the matching result of the first characteristic information, the matching result of the second characteristic information and the matching result of the third characteristic information based on the weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, so as to generate a recognition result of the user to be recognized.

In a seventh aspect, embodiments of the present application provide an apparatus for generating a gait recognition model, the apparatus comprising: a sample acquisition unit configured to acquire a set of training samples, wherein the training samples include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum being wireless spectrums that characterize gait of the same user, the first wireless spectrum and the third wireless spectrum being wireless spectrums that characterize gait of different users; and the model training unit is configured to train to obtain a gait recognition model based on the training sample set and a preset loss function.

In some embodiments, the model training unit comprises: a training module configured to perform the training steps of: for training samples in the training sample set, respectively inputting a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum of the training samples into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless frequency spectrum, second wireless frequency spectrum and third wireless frequency spectrum; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained first characteristic information set, second characteristic information set and third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, the model training unit further comprises: the adjusting module is configured to adjust parameters of the initial convolutional neural network if the value of the loss function is larger than a preset value, and the training step is continuously executed by using the adjusted initial convolutional neural network as the initial convolutional neural network.

In an eighth aspect, embodiments of the present application provide an apparatus for recognizing gait, the apparatus comprising: an information acquisition unit configured to acquire a wireless spectrum characterizing gait of a user to be identified; the identification unit is configured to input a wireless frequency spectrum into a gait recognition model generated by adopting the method described by any implementation manner of the first aspect to obtain characteristic information of a user to be recognized; a matching unit configured to match the feature information with pre-stored feature information; and a result generation unit configured to generate a recognition result of the user to be recognized based on the matching result.

In a ninth aspect, embodiments of the present application provide an apparatus for generating a gait recognition model, the gait recognition model including a first sub-recognition model and a second sub-recognition model, the apparatus comprising: a first sample acquisition unit configured to acquire a first training sample set, wherein the first training sample includes a first gait energy diagram, a second gait energy diagram, and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram being gait energy diagrams that characterize the gait of the same user, the first gait energy diagram and the third gait energy diagram being gait energy diagrams that characterize the gait of different users; a second sample acquisition unit configured to acquire a second set of training samples, wherein the second training samples include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum being wireless spectrums that characterize gait of the same user, the first wireless spectrum and the third wireless spectrum being wireless spectrums that characterize gait of different users; the first model training unit is configured to train to obtain a first sub-recognition model based on the first training sample set and a preset loss function; the second model training unit is configured to train to obtain a second sub-recognition model based on the second training sample set and the loss function; the weight determining unit is configured to determine weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In some embodiments, the first model training unit comprises: a first training module configured to perform a first training step of: for a first training sample in a first training sample set, respectively inputting a first gait energy diagram, a second gait energy diagram and a third gait energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first gait energy diagram, second gait energy diagram and third gait energy diagram; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained first characteristic information set, second characteristic information set and third characteristic information set; and if the value of the loss function is smaller than or equal to a preset numerical value, determining the initial convolutional neural network as a first sub-recognition model.

In some embodiments, the first model training unit further comprises: the first adjustment module is configured to adjust parameters of the initial convolutional neural network if the value of the loss function is greater than a preset value, and continuously execute the first training step by using the adjusted initial convolutional neural network as the initial convolutional neural network.

In some embodiments, the second model training unit comprises: a second training module configured to perform a second training step of: for a second training sample in the second training sample set, respectively inputting a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum of the second training sample into an initial convolutional neural network to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless frequency spectrum, second wireless frequency spectrum and third wireless frequency spectrum; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained fourth characteristic information set, fifth characteristic information set and sixth characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some embodiments, the second model training unit further comprises: and the second adjusting module is configured to adjust parameters of the initial convolutional neural network if the value of the loss function is larger than a preset value, and continuously execute the second training step by using the adjusted initial convolutional neural network as the initial convolutional neural network.

In a tenth aspect, embodiments of the present application provide an apparatus for recognizing gait, the apparatus comprising: an information acquisition unit configured to acquire a gait energy diagram and a wireless spectrum characterizing a gait of a user to be identified; a first recognition unit configured to input a gait energy diagram to a first sub-recognition model of a gait recognition model generated by the method described in any implementation manner of the third aspect to obtain first feature information of a user to be recognized; a second recognition unit configured to input a wireless spectrum to a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized; a first matching unit configured to match the first characteristic information with pre-stored first characteristic information; a second matching unit configured to match the second characteristic information with pre-stored second characteristic information; and the result generating unit is configured to fuse the matching result of the first characteristic information and the matching result of the second characteristic information based on the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generate a recognition result of the user to be recognized.

In an eleventh aspect, embodiments of the present application provide an apparatus for generating a gait recognition model, the gait recognition model including a first sub-recognition model, a second sub-recognition model, and a third sub-recognition model, the apparatus comprising: a first sample acquisition unit configured to acquire a first training sample set, wherein the first training sample includes a first gait energy diagram, a second gait energy diagram, and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram being gait energy diagrams that characterize the gait of the same user, the first gait energy diagram and the third gait energy diagram being gait energy diagrams that characterize the gait of different users; the second sample acquisition unit is configured to acquire a second training sample set, wherein the second training sample comprises a first Wi-Fi frequency spectrum, a second Wi-Fi frequency spectrum and a third Wi-Fi frequency spectrum, the first Wi-Fi frequency spectrum and the second Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing the gait of the same user, and the first Wi-Fi frequency spectrum and the third Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing the gait of different users; a third sample acquisition unit configured to acquire a third training sample set, wherein the third training sample includes a first millimeter wave spectrum, a second millimeter wave spectrum, and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectrums representing gait of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectrums representing gait of different users; a first model training unit is provided for the first model, training to obtain a first sub-recognition model based on the first training sample set and a preset loss function; the second model training unit is configured to train to obtain a second sub-recognition model based on the second training sample set and the loss function; a third model training unit configured to train to obtain a third sub-recognition model based on the third training sample set and the loss function; and the weight determining unit is configured to determine weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set and the number of the third training samples in the third training sample set.

In a twelfth aspect, embodiments of the present application provide an apparatus for recognizing gait, the apparatus comprising: an information acquisition unit configured to acquire a gait energy diagram, wi-Fi spectrum, and millimeter wave spectrum that characterize gait of a user to be identified; a first recognition unit configured to input a gait energy diagram to a first sub-recognition model of the gait recognition model generated by the method described in the fifth aspect to obtain first feature information of a user to be recognized; the second recognition unit is configured to input Wi-Fi frequency spectrums into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized; the third recognition unit is configured to input the millimeter wave frequency spectrum into a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized; a first matching unit configured to match the first characteristic information with pre-stored first characteristic information; a second matching unit configured to match the second characteristic information with pre-stored second characteristic information; a third matching unit configured to match the third characteristic information with pre-stored third characteristic information; the result generating unit is configured to fuse the matching result of the first characteristic information, the matching result of the second characteristic information and the matching result of the third characteristic information based on the weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, and generate a recognition result of the user to be recognized.

In a thirteenth aspect, embodiments of the present application provide an electronic device, including: one or more processors; a storage device having one or more programs stored thereon; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described in any of the implementations of the first to sixth aspects.

In a fourteenth aspect, embodiments of the present application provide a computer readable medium having stored thereon a computer program which, when executed by a processor, implements a method as described in any one of the first to sixth aspects.

According to the method and the device for generating the gait recognition model, the training sample set comprising the training samples generated by the wireless spectrum representing the gait of the same user and the wireless spectrum representing the gait of different users is obtained, and then the gait recognition model is obtained based on the training sample set and the preset loss function training, so that the accuracy of gait recognition can be improved.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, made with reference to the following drawings, in which:

FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

FIG. 2 is a flow chart of one embodiment of a method for generating a gait recognition model in accordance with the present application;

FIG. 3 is a flow chart of one embodiment of a method for recognizing gait in accordance with the present application;

FIG. 4 is a flow chart of another embodiment of a method for generating a gait recognition model in accordance with the present application;

FIG. 5 is a flow chart of another embodiment of a method for recognizing gait in accordance with the application;

FIG. 6 is a flow chart of yet another embodiment of a method for generating a gait recognition model in accordance with the present application;

FIG. 7 is a flow chart of yet another embodiment of a method for recognizing gait in accordance with the application;

FIGS. 8 and 9 are schematic diagrams of one application scenario of a method for generating a gait recognition model in accordance with the present application;

FIG. 10 is a schematic structural view of one embodiment of an apparatus for generating a gait recognition model in accordance with the present application;

FIG. 11 is a schematic structural view of one embodiment of a device for recognizing gait in accordance with the present application;

FIG. 12 is a schematic structural view of another embodiment of an apparatus for generating a gait recognition model in accordance with the present application;

FIG. 13 is a schematic structural view of another embodiment of a device for recognizing gait in accordance with the application;

FIG. 14 is a schematic structural view of yet another embodiment of an apparatus for generating a gait recognition model in accordance with the present application;

FIG. 15 is a schematic structural view of yet another embodiment of a device for recognizing gait in accordance with the present application;

fig. 16 is a schematic diagram of a computer system suitable for use in implementing embodiments of the present application.

Detailed Description

The present application is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for convenience of description, only the portions relevant to the relevant invention are shown in the drawings.

It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

FIG. 1 shows a method for generating a gait recognition model, a method for recognizing gait, a program, a computer-readable medium, and a computer-readable medium, to which the present application can be applied an example system architecture 100 of an embodiment of a device for generating a gait recognition model or a device for recognizing gait.

As shown in fig. 1, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and servers 105, 106. The network 104 is used for the communication between the terminal devices 101, 102, 103 and the server 105 106 provides a medium for a communication link therebetween. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the servers 105, 106 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as gait recognition type applications, web browser applications, shopping type applications, search type applications, instant messaging tools, mailbox clients, social platform software, etc., may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices supporting gait recognition, including but not limited to smartphones, tablets, laptop portable computers, desktop computers, and the like. When the terminal devices 101, 102, 103 are software, they can be installed in the above-listed electronic devices. Which may be implemented as multiple software or software modules (e.g., to provide distributed services), or as a single software or software module. The present invention is not particularly limited herein.

Server 105 may be a server that provides various services, such as a data server that stores training samples. The data server may store a first set of training samples. The first training samples may include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum. The first wireless spectrum and the second wireless spectrum are wireless spectrums for representing the gait of the same user, and the first wireless spectrum and the third wireless spectrum are wireless spectrums for representing the gait of different users.

Server 105 may also store a second set of training samples. The second training sample may include a first gait energy pattern, a second gait energy pattern, and a third gait energy pattern. The first gait energy diagram and the second gait energy diagram are gait energy diagrams representing the gait of the same user, and the first gait energy diagram and the third gait energy diagram are gait energy diagrams representing the gait of different users.

The server 106 may be a server providing various services, such as a background server providing support for gait recognition type applications on the terminal devices 101, 102, 103. The background server may train the model to be trained to obtain a gait recognition model using the training sample set stored in the data server 105. The background server can also input information (such as wireless frequency spectrum) representing the gait of the user to be identified, which is submitted by the terminal equipment, into the gait recognition model to obtain the characteristic information of the user to be identified, and generate a recognition result by utilizing the characteristic information output by the model and the pre-stored characteristic information.

It should be noted that the method for generating a gait recognition model or the method for recognizing a gait provided in the embodiments of the present application is generally performed by the server 106, and accordingly, the device for generating a gait recognition model or the device for recognizing a gait is generally provided in the server 106.

The servers 105 and 106 may be hardware or software. When the servers 105 and 106 are hardware, the servers may be realized as a distributed server cluster composed of a plurality of servers, or may be realized as a single server. When the server is software, it may be implemented as a plurality of software or software modules (e.g., to provide distributed services), or as a single software or software module. The present invention is not particularly limited herein.

It should be noted that the local server 106 may also directly store the training sample set, and the server 106 may directly obtain the local training sample set. At this point, the exemplary system architecture 100 may not include the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any suitable number of terminal devices, networks, and servers, as desired for implementation.

With continued reference to fig. 2, a flow 200 of one embodiment of a method for generating a gait recognition model in accordance with the present application is shown. The method for generating a gait recognition model may comprise the steps of:

in step 201 the process is carried out, a training sample set is obtained.

In this embodiment, the subject performing the method for generating a gait recognition model (e.g., the server 106 of fig. 1) may obtain the training sample set locally or remotely. Wherein the training samples may include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum. The first and second wireless spectrums are wireless spectrums that characterize gait of the same user, and the first and third wireless spectrums are wireless spectrums that characterize gait of different users. As an example, in a certain training sample, the first and second wireless spectrum may characterize the gait information of the sample user "Zhang Sanj", and the third wireless spectrum may characterize the gait information of the sample user "Liqu".

In some alternative implementations of the present embodiment, the first, second, and third wireless spectrum may be selected from a pre-stored set of wireless spectrum (e.g., a set of wireless spectrum pre-stored on a server).

Corresponding to this implementation, the radio spectrum of the set of radio spectrum may be generated by:

firstly, during the period that a sample user walks along a preset route, wireless signal acquisition is carried out on a target position of an area where the sample user is located, and a wireless signal described by CSI (Channel State Information ) is obtained. Here, the target position may be a specific position located on the wireless signal propagation path. CSI describes signal changes in the wireless signal from transmission to reception, and human body characteristics and movements change CSI description information.

Alternatively, the Wireless signal may include a Wi-Fi (Wireless-Fidelity) signal or a millimeter wave signal. In one example, wi-Fi signals may be acquired using a wireless router, and then the acquired Wi-Fi signals are described with CSI. In another example, millimeter wave signals may be acquired using millimeter wave transmitting/receiving devices, and then the acquired millimeter wave signals may be described using CSI.

Then, denoising the acquired wireless signals. As an example, PCA (Principal Component Analysis ) techniques or other suitable techniques may be used to remove noise from the CSI-described wireless signal, improving the accuracy of the wireless signal.

Finally, performing time-frequency conversion on the denoised wireless signals to obtain a wireless spectrum representing the gait of the sample user. As an example, the denoised wireless signal may be subjected to a short-time fourier transform (Short Time Fourier Transform) that converts the wireless signal from the time domain to the frequency domain, resulting in a wireless spectrum that characterizes the gait of the sample user.

In this implementation, the execution subject of the wireless-spectrum generation step may be the same as or different from the execution subject of the method for generating the gait recognition model. If so, the executing body of the radio spectrum generation step may store the radio spectrum set locally after generating the radio spectrum set. If it is different from the one in the above, the execution body of the wireless-spectrum generating step may transmit the wireless-spectrum set to the execution body of the method for generating the gait recognition model after generating the wireless-spectrum set.

Step 202, training to obtain a gait recognition model based on the training sample set and a preset loss function.

In this embodiment, the subject of execution of the method for generating a gait recognition model (e.g., the server 106 of figure 1) may use a training sample set and a preset loss function, training the initial convolutional neural network by using a machine learning method to obtain a gait recognition model. Here, the initial convolutional neural network may use various convolutional neural network (Convolutional Neural Network, CNN) structures existing. For example DenseNet, googleNet, VGGNet, resNet, etc.

In some optional implementations of the present embodiment, step 202 may specifically include:

first, the following training steps are performed:

first, for each training sample in the training sample set, a first wireless spectrum, a second wireless spectrum and a third wireless spectrum of the training sample are respectively input into an initial convolutional neural network, so as to obtain first characteristic information, second characteristic information and third characteristic information corresponding to the input first wireless spectrum, second wireless spectrum and third wireless spectrum. Here, the first, second, and third characteristic information may be information characterizing gait of the human body, and may be represented by a vector or a matrix.

Then, based on the obtained first feature information set, second feature information set, and third feature information set, it is determined whether a value of a preset loss function (for example, a value of a loss function that can be obtained by inputting each of the first feature information, second feature information, and third feature information into the preset loss function) is less than or equal to a preset numerical value.

If the value of the loss function is smaller than or equal to the preset value, the initial convolutional neural network is determined to be a gait recognition model.

And secondly, if the value of the loss function is larger than a preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuously executing the training step.

In the training process of the initial convolutional neural network, the partial derivatives of the loss function to the weights of the neurons can be calculated layer by layer to form the gradient of the loss function to the weight vector so as to modify the weights of the initial convolutional neural network. The learning of the gait recognition model is completed in the weight modification process. When the value of the loss function reaches a desired value (e.g., less than 0.05), training of the gait recognition model is completed.

It should be noted that the objective of training the initial convolutional neural network is to make the differences in the characteristic information extracted from the wireless spectrum characterizing the gait of the same user as small as possible, at the same time, the difference in the feature information extracted from the wireless spectrum characterizing the gait of different users is made as large as possible. Here, the differences in the feature information may be characterized by the similarity (e.g., euclidean distance, cosine similarity, etc.) of the feature information. Here, the difference in the characteristic information extracted from the wireless spectrum characterizing the gait of the same user may be referred to as a first difference, and the difference in the characteristic information extracted from the wireless spectrum characterizing the gait of different users may be referred to as a second difference. The Loss function may be a function, such as a Triplet Loss function (Triplet Loss), that characterizes the degree of difference of the second difference from the first difference. As an example, a Triplet Loss can be defined as:

Where N is a natural number greater than 1, i is the number of training samples, i= {1,2, …, N },first, second and third radio spectrum, respectively, of the ith training sample,/->For the first wireless spectrum->Corresponding first characteristic information, < >>For the second radio spectrum->Corresponding second characteristic information, < >>For the third wireless spectrum->And corresponding third characteristic information, wherein alpha is a constant.

In this embodiment, a triplet composed of two wireless spectrums representing the gait of the same user and two wireless spectrums representing the gait of different users is used as a training sample, and the trained gait recognition model is not affected by environmental light, obstacles and the like, so that the accuracy of gait recognition is improved.

According to the method for generating the gait recognition model, the training sample set comprising the training samples generated by the wireless spectrum representing the gait of the same user and the wireless spectrum representing the gait of different users is obtained, and then the gait recognition model is obtained based on the training sample set and the preset loss function training, so that the accuracy of gait recognition can be improved.

With further reference to fig. 3, a flow 300 of one embodiment of a method for recognizing gait in accordance with the present application is shown. The method for recognizing gait may include the steps of:

Step 301, a wireless spectrum characterizing the gait of a user to be identified is acquired.

In this embodiment, the subject performing the method for recognizing gait (e.g., the server 106 of fig. 1) may acquire a wireless spectrum for characterizing the gait of the user to be recognized.

In some optional implementations of this embodiment, step 301 may specifically include:

first, during the user to be identified walks along a preset route, and acquiring a wireless signal at a target position of the area where the user to be identified is located, and obtaining the wireless signal described by the CSI. Here the number of the elements is the number, the wireless signals may include Wi-Fi signals or millimeter wave signals.

And secondly, denoising the acquired wireless signals.

Finally, performing time-frequency conversion on the denoised wireless signals to obtain a wireless frequency spectrum representing the gait of the user to be identified.

It should be noted that the wireless spectrum for characterizing the gait of the user to be identified may also be acquired by a terminal device or a server other than the execution subject and then transmitted to the execution subject.

Step 302, inputting the wireless spectrum into a gait recognition model to obtain the characteristic information of the user to be recognized.

In this embodiment, the execution subject (e.g., the server 106 in fig. 1) of the method for recognizing gait may input the wireless spectrum acquired in step 301 into a trained gait recognition model to obtain the feature information of the user to be recognized. Wherein, the gait recognition model may be generated using the method described in the corresponding embodiment of figure 2. The characteristic information may be information characterizing human gait, and may be represented by a vector or a matrix.

Step 303, the feature information is matched with the pre-stored feature information.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the above-described characteristic information with pre-stored characteristic information. Wherein the pre-stored characteristic information may be pre-extracted from a wireless spectrum characterizing the gait of the user. The execution body may determine the similarity between the feature information and the pre-stored feature information (e.g., may determine using euclidean distance, cosine similarity, etc.). If the similarity is greater than or equal to a preset value, the feature information can be determined to be matched with the pre-stored feature information. If the similarity is smaller than the preset value, the fact that the characteristic information is not matched with the pre-stored characteristic information can be determined.

Step (a) 304, and generating a recognition result of the user to be recognized based on the matching result.

In this embodiment, an execution subject of the method for recognizing gait (e.g., the server 106 of fig. 1) may generate the recognition result of the user to be recognized based on the matching result. For example, if the feature information is matched with the pre-stored feature information, a recognition result indicating that the identity authentication passes may be generated; if the characteristic information is not matched with the pre-stored characteristic information, an identification result indicating that the identity authentication fails can be generated.

According to the method for identifying gait, the acquired wireless spectrum for representing the gait of the user to be identified is input into the trained gait identification model, then the characteristic information output by the model is matched with the pre-stored characteristic information, and finally the identification result of the user to be identified is generated based on the matching result, so that the accuracy of gait identification can be improved.

With further reference to fig. 4, a flow 400 of another embodiment of a method for generating a gait recognition model in accordance with the present application is shown. The method for generating a gait recognition model may comprise the steps of:

step 401, a first set of training samples is obtained.

In this embodiment, the subject performing the method for generating a gait recognition model (e.g., the server 106 of fig. 1) may obtain the first set of training samples locally or remotely. The first training sample may include a first gait energy pattern, a second gait energy pattern, and a third gait energy pattern. The first and second gait energy patterns are gait energy patterns that characterize the gait of the same user, and the first and third gait energy patterns are gait energy patterns that characterize the gait of different users.

The gait energy diagram (Gait Engery Image, GEI) is a common feature in gait detection, and the extraction method is simple, and can well represent the features of speed, morphology and the like of the gait. As an example, the gait energy diagram may be defined as:

where x, y are pixel coordinates, M is the total number of key frames, t is the number of key frames, t= {1,2, …, M }, G (x, y) represents the final gait energy map, which is a two-dimensional image of x and y, I (x, y, t) is the pixel value (e.g., gray value) of the pixel with coordinates (x, y) in the key frame numbered t.

In some alternative implementations of the present embodiments, the first, second, and third gait energy patterns may be selected from a pre-stored set of gait energy patterns.

Corresponding to this implementation, the gait energy patterns in the set of gait energy patterns may be generated by:

firstly, during the walking period of a sample user along a preset route, video acquisition is carried out on the sample user, and a walking video of the sample user is obtained. For example, a camera may be used to capture a video of the walking of a sample user.

Thereafter, a plurality of key frames are extracted from the walking video. As an example, the required key frames may be extracted from the walking video in 10 frames per second.

Then, the human body contour of the sample user is extracted from the plurality of key frames, and a human body silhouette (also referred to as gait silhouette) is generated. For example, a human segmentation method may be used to extract moving objects (i.e., sample users) from key frames.

Finally, a gait energy map is synthesized based on the generated gait silhouette.

In the present embodiment, the execution subject of the gait energy pattern generation step may be the same as or different from the execution subject of the method for generating the gait recognition model. If the same, the execution subject of the gait energy pattern generation step may store the gait energy pattern set locally after generating the gait energy pattern set. If different, the execution body of the gait energy pattern generation step may transmit the gait energy pattern set to the execution body of the method for generating a gait recognition model after generating the gait energy pattern set.

Step 402, a second set of training samples is obtained.

In this embodiment, the subject performing the method for generating a gait recognition model (e.g., the server 106 of fig. 1) may obtain the second set of training samples locally or remotely. Wherein the second training samples may include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum. The first and second wireless spectrums are wireless spectrums that characterize gait of the same user, and the first and third wireless spectrums are wireless spectrums that characterize gait of different users.

In some optional implementations of the present embodiment, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum may be selected from a set of pre-stored wireless spectrums. The step of generating the radio spectrum in the radio spectrum set may refer to the description of the step of generating the radio spectrum in the corresponding embodiment of fig. 2, which is not described herein.

In some alternative implementations of the present embodiment, the wireless signals include Wi-Fi signals or millimeter wave signals.

Step 403, training to obtain a first sub-recognition model based on the first training sample set and a preset loss function.

In this embodiment, the execution body (e.g., the server 106 of fig. 1) of the method for generating the gait recognition model may train the initial convolutional neural network by using the machine learning method using the first training sample set and the preset loss function to obtain the first sub-recognition model. Here, the initial convolutional neural network may use various convolutional neural network structures existing.

In some optional implementations of this embodiment, step 403 may specifically include:

first, the following first training steps are performed:

first, for each first training sample in the first training sample set, a first gait energy diagram, a second gait energy diagram and a third gait energy diagram of the first training sample are respectively input into an initial convolutional neural network, and first characteristic information, second characteristic information and third characteristic information corresponding to the input first gait energy diagram, second gait energy diagram and third gait energy diagram are obtained. Here, the first, second, and third characteristic information may be information characterizing gait of the human body, and may be represented by a vector or a matrix.

If the value of the loss function is smaller than or equal to a preset numerical value, the initial convolutional neural network is determined to be a first sub-recognition model.

Step 404, training to obtain a second sub-recognition model based on the second training sample set and the loss function.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for generating the gait recognition model may train the initial convolutional neural network using the machine learning method using the second training sample set and the preset loss function to obtain the second sub-recognition model. Here, the initial convolutional neural network may use various convolutional neural network structures existing.

In some alternative implementations of the present embodiment, step 404 may specifically include:

the first step, the following second training step is performed:

first, for each second training sample in the second training sample set, the first wireless spectrum, the second wireless spectrum and the third wireless spectrum of the second training sample are respectively input into an initial convolutional neural network, so as to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless spectrum, second wireless spectrum and third wireless spectrum. Here, the fourth characteristic information, the fifth characteristic information, and the sixth characteristic information may be information characterizing the gait of the human body, and may be represented by vectors or matrices.

Then, based on the obtained fourth, fifth, and sixth feature information sets, it is determined whether the value of the preset loss function (for example, the value of the loss function that can be obtained by inputting each of the fourth, fifth, and sixth feature information into the preset loss function) is less than or equal to a preset numerical value.

And if the value of the loss function is smaller than or equal to the preset value, determining the initial convolutional neural network as a second sub-recognition model.

And a second step of adjusting parameters of the initial convolutional neural network if the value of the loss function is larger than a preset value, and continuously executing the second training step by using the adjusted initial convolutional neural network as the initial convolutional neural network.

In this embodiment, the first sub-recognition model and the second sub-recognition model may be trained using the same initial convolutional neural network and the loss function, the first sub-recognition model and the second sub-recognition model may also be trained using different initial convolutional neural networks and loss functions, which are not specifically limited in this application.

Step 405, determining weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for generating the gait recognition model may determine weights in the output of the gait recognition model for the model output of the first sub-recognition model and the model output of the second sub-recognition model, respectively, according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set. As an example, if the number of the first training samples in the first training sample set is 3000 and the number of the second training samples in the second training sample set is 1000, it may be determined that the weight ratio of the model output of the first sub-recognition model to the model output of the second sub-recognition model in the output of the gait recognition model is 3:1.

As can be seen in fig. 4, the flow 400 of the method for generating a gait recognition model in this embodiment embodies the step of training a first sub-recognition model of the gait recognition model using the gait energy diagram, as compared to the corresponding embodiment of fig. 2. Therefore, the scheme described in the embodiment can make up for application scenes (for example, different walking routes, different wearing and the like) with low recognition rate of the gait recognition model trained by using the wireless spectrum, so that the accuracy rate of gait recognition is further improved.

With further reference to fig. 5, a flow 500 of another embodiment of a method for recognizing gait in accordance with the present application is shown. The method for recognizing gait may include the steps of:

step 501, a gait energy diagram and a wireless spectrum are acquired for characterizing the gait of a user to be identified.

In this embodiment, the subject (e.g., server 106 of fig. 1) performing the method for recognizing gait may acquire a gait energy map and a wireless spectrum for characterizing the gait of the user to be recognized.

In some alternative implementations of the present embodiment, step 501 may specifically include:

firstly, during the walking of a user to be identified along a preset route, video acquisition is carried out on the user to be identified, and a walking video of the user to be identified is obtained. For example, a camera may be used to capture a video of walking of the user to be identified.

Then, the human body contour of the user to be identified is extracted from the plurality of key frames, and a human body silhouette (also referred to as gait silhouette) is generated. For example, a moving object (i.e., a user to be identified) may be extracted from the key frames using a human segmentation method.

In some optional implementations of the present embodiment, step 501 may further include:

firstly, during the walking of a user to be identified along a preset route, wireless signal acquisition is carried out on a target position of an area where the user to be identified is located, and a wireless signal described by CSI is obtained. Here, the wireless signal may include a Wi-Fi signal or a millimeter wave signal.

And secondly, denoising the acquired wireless signals.

The gait energy pattern and the wireless spectrum for characterizing the gait of the user to be identified may be acquired by a terminal device or a server other than the execution subject and then transmitted to the execution subject.

Step 502, inputting the gait energy diagram to a first sub-recognition model of the gait recognition model to obtain first characteristic information of the user to be recognized.

In this embodiment, the execution body (e.g., the server 106 of fig. 1) of the method for recognizing gait may input the gait energy diagram obtained in step 501 into the first sub-recognition model of the trained gait recognition model, to obtain the first feature information of the user to be recognized. The gait recognition model may be generated by the method described in the corresponding embodiment of fig. 4. The first characteristic information may be information characterizing human gait, and may be represented by a vector or a matrix.

Step 503, inputting the wireless spectrum to a second sub-recognition model of the gait recognition model, to obtain second feature information of the user to be recognized.

In this embodiment, the execution subject (e.g., the server 106 in fig. 1) of the method for gait recognition may input the wireless spectrum acquired in step 501 into the second sub-recognition model of the trained gait recognition model, to obtain the second feature information of the user to be recognized. The second characteristic information may be information representing gait of the human body, and may be represented by a vector or a matrix.

Step 504, the first characteristic information is matched with the pre-stored first characteristic information.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the first characteristic information described above with the pre-stored first characteristic information. Wherein the pre-stored first characteristic information may be pre-extracted from a gait energy pattern characterizing the gait of the user. Here, the similarity between the first feature information and the pre-stored first feature information may be determined by matching the first feature information and the pre-stored first feature information (for example, may be determined by using euclidean distance, cosine similarity, or the like).

Step 505, the second characteristic information is matched with the pre-stored second characteristic information.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the second characteristic information described above with the pre-stored second characteristic information. Wherein the pre-stored second characteristic information may be pre-extracted from a wireless spectrum characterizing the gait of the user. Here, the similarity between the second feature information and the pre-stored second feature information may be determined (for example, may be determined using euclidean distance, cosine similarity, or the like).

Step 506, based on the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, the matching result of the first feature information and the matching result of the second feature information are fused, and the recognition result of the user to be recognized is generated.

In this embodiment, the execution body (e.g., the server 106 in fig. 1) of the method for recognizing gait may fuse the matching result of the first feature information and the matching result of the second feature information to generate the recognition result of the user to be recognized. As an example, the weight ratio of the model output of the first sub-recognition model to the model output of the second sub-recognition model in the output of the gait recognition model is 3:1, the similarity of the first characteristic information output by the first sub-recognition model to the pre-stored first characteristic information is 90%, the similarity of the second characteristic information output by the second sub-recognition model to the pre-stored second characteristic information is 75%, and the recognition result of the user to be recognized may be 90% x 0.75+75% x 0.25=86.25%. If the similarity threshold is 85%, the identification result can indicate that the identity authentication of the user to be identified passes.

As can be seen from fig. 5, compared with the embodiment corresponding to fig. 3, the flow 500 of the method for recognizing gait in this embodiment represents the steps of acquiring the gait energy diagram and the wireless spectrum of the user to be recognized and fusing their recognition results to generate the recognition result. Therefore, the scheme described in the embodiment can make up for application scenes (for example, different walking routes, different wearing and the like) with low recognition rate of the gait recognition model trained by using the wireless spectrum, so that the accuracy rate of gait recognition is further improved.

With further reference to fig. 6, a flow 600 of yet another embodiment of a method for generating a gait recognition model in accordance with the present application is shown. The method for generating a gait recognition model may comprise the steps of:

step 601, a first set of training samples is obtained.

Step 602, a second set of training samples is obtained.

In this embodiment, the subject performing the method for generating a gait recognition model (e.g., the server 106 of fig. 1) may obtain the second set of training samples locally or remotely. The second training samples may include a first Wi-Fi spectrum, a second Wi-Fi spectrum, and a third Wi-Fi spectrum. The first Wi-Fi spectrum and the second Wi-Fi spectrum are Wi-Fi spectra characterizing gait of the same user, and the first Wi-Fi spectrum and the third Wi-Fi spectrum are Wi-Fi spectra characterizing gait of different users.

Step 603, obtaining a third training sample set.

In this embodiment, the subject performing the method for generating a gait recognition model (e.g., the server 106 of fig. 1) may obtain the third set of training samples locally or remotely. Wherein the third training sample may include a first millimeter wave spectrum, a second millimeter wave spectrum, and a third millimeter wave spectrum. The first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectrums for representing gait of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectrums for representing gait of different users.

Step 604, training to obtain a first sub-recognition model based on the first training sample set and a preset loss function.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for generating the gait recognition model may train the initial convolutional neural network by using the machine learning method using the first training sample set and the preset loss function to obtain the gait recognition model. Here, the initial convolutional neural network may use various convolutional neural network structures existing.

The specific training steps of the first sub-recognition model may refer to the description of the training steps of the first sub-recognition model in the corresponding embodiment of fig. 4, which is not described herein.

Step 605, training to obtain a second sub-recognition model based on the second training sample set and the loss function.

Step 606, training to obtain a third sub-recognition model based on the third training sample set and the loss function.

In this embodiment, the execution body (e.g., the server 106 in fig. 1) of the method for generating the gait recognition model may train the initial convolutional neural network by using the machine learning method using the third training sample set and the preset loss function, to obtain the third sub-recognition model. Here, the initial convolutional neural network may use various convolutional neural network structures existing.

The specific training steps of the second sub-recognition model and the third sub-recognition model may refer to the description of the training steps of the second sub-recognition model in the corresponding embodiment of fig. 4, which is not described herein.

In step 607, the weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model, and the model output of the third sub-recognition model in the output of the gait recognition model are determined according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set, and the number of the third training samples in the third training sample set.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for generating a gait recognition model may determine weights in the output of the gait recognition model for the model output of the first sub-recognition model, the model output of the second sub-recognition model, and the model output of the third sub-recognition model, respectively, according to the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set, and the number of the third training samples in the third training sample set. As an example, if the number of the first training samples in the first training sample set is 3000, the number of the second training samples in the second training sample set is 1000, and the number of the third training samples in the third training sample set is 1000, it may be determined that the weight ratio of the model outputs of the first sub-recognition model, the second sub-recognition model, and the third sub-recognition model in the output of the gait recognition model is 3:1:1.

As can be seen from fig. 6, compared to the corresponding embodiment of fig. 2, the flow 600 of the method for generating a gait recognition model in this embodiment embodies the steps of training the first, second and third sub-recognition models of the gait recognition model using the gait energy diagram, wi-Fi spectrum and millimeter wave spectrum. Therefore, the scheme described in the embodiment can make up for application scenes (for example, different walking routes, different wearing and the like) with low recognition rate of the gait recognition model trained by using a single wireless frequency spectrum, so that the accuracy rate of gait recognition is further improved.

With further reference to fig. 7, a flow 700 of yet another embodiment of a method for recognizing gait in accordance with the present application is shown. The method for recognizing gait may include the steps of:

step 701, acquiring a gait energy diagram, wi-Fi spectrum and millimeter wave spectrum for characterizing the gait of a user to be identified.

In this embodiment, the executing subject of the method for recognizing gait (e.g., the server 106 of fig. 1) may acquire a gait energy map Wi-Fi spectrum and a millimeter wave spectrum for characterizing the gait of the user to be recognized.

Step 702, inputting the gait energy diagram to a first sub-recognition model of the gait recognition model, to obtain first feature information of the user to be recognized.

In this embodiment, the execution body (e.g., the server 106 of fig. 1) of the method for recognizing gait may input the acquired gait energy diagram into the first sub-recognition model of the trained gait recognition model, to obtain the first feature information of the user to be recognized. The gait recognition model may be generated by the method described in the corresponding embodiment of fig. 6. The first characteristic information may be information characterizing human gait, and may be represented by a vector or a matrix.

And step 703, inputting the Wi-Fi frequency spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may input the acquired Wi-Fi spectrum into the second sub-recognition model of the trained gait recognition model, to obtain the second feature information of the user to be recognized. The second characteristic information may be information representing gait of the human body, and may be represented by a vector or a matrix.

Step 704, inputting the millimeter wave spectrum to a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may input the acquired millimeter wave spectrum into the third sub-recognition model of the trained gait recognition model, to obtain the third feature information of the user to be recognized. The third characteristic information may be information representing gait of the human body, and may be represented by a vector or a matrix.

Step 705, the first characteristic information is matched with the pre-stored first characteristic information.

Step 706, the second characteristic information is matched with the pre-stored second characteristic information.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the second characteristic information described above with the pre-stored second characteristic information. Wherein the pre-stored second characteristic information may be pre-extracted from Wi-Fi spectrum characterizing the gait of the user. Here, the similarity between the second feature information and the pre-stored second feature information may be determined (for example, may be determined using euclidean distance, cosine similarity, or the like).

Step 707, matching the third characteristic information with the pre-stored third characteristic information.

In this embodiment, the execution subject (e.g., the server 106 of fig. 1) of the method for recognizing gait may match the third characteristic information described above with the pre-stored third characteristic information. The pre-stored third characteristic information may be pre-extracted from a millimeter wave spectrum characterizing gait of the user. Here, the similarity between the third feature information and the pre-stored third feature information may be determined (for example, may be determined using euclidean distance, cosine similarity, or the like).

Step 708, based on the weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, the matching result of the first feature information, the matching result of the second feature information and the matching result of the third feature information are fused, and the recognition result of the user to be recognized is generated.

In this embodiment, the execution body (e.g., the server 106 in fig. 1) of the method for recognizing gait may fuse the matching result of the first feature information, the matching result of the second feature information, and the matching result of the third feature information to generate the recognition result of the user to be recognized. As an example, the weight ratio of the model outputs of the first sub-recognition model, the second sub-recognition model, and the third sub-recognition model in the output of the gait recognition model is 3:1:1, the similarity of the first feature information output by the first sub-recognition model to the pre-stored first feature information is 90%, the similarity of the second feature information output by the second sub-recognition model to the pre-stored second feature information is 75%, the similarity of the third feature information output by the third sub-recognition model to the pre-stored third feature information is 87%, and the recognition result of the user to be recognized may be 90% x 0.6+75% x 0.2+87% x 0.2=86.4%. If the similarity threshold is 85%, the identification result can indicate that the identity authentication of the user to be identified passes.

As can be seen from fig. 7, compared with the embodiment corresponding to fig. 3, the flow 700 of the method for recognizing gait in this embodiment represents the steps of acquiring the gait energy diagram, wi-Fi spectrum and millimeter wave spectrum of the user to be recognized and fusing the recognition results thereof to generate the recognition result. Therefore, the scheme described in the embodiment can make up for application scenes (for example, different walking routes, different wearing and the like) with low recognition rate of the gait recognition model trained by using a single wireless frequency spectrum, so that the accuracy rate of gait recognition is further improved.

With continued reference to fig. 8 and 9, fig. 8 and 9 are schematic diagrams of one application scenario of a method for generating a gait recognition model according to the present application.

As shown in fig. 8, first, three preset routes L are provided in a room shown by a solid line frame ₁ ～L ₃ Three cameras 802-804, a wireless router 801, a radio frequency receiving end 805, and a radio frequency transmitting end 806. After that, 50 experimenters 807 are invited to follow the preset route L, respectively ₁ ～L ₃ Walking, collecting walking videos of experimenters 807 through cameras 802-804, collecting Wi-Fi signals of each experimenter during walking through a wireless router 801, and collecting millimeter wave signals emitted by a millimeter wave radio frequency emitting end 806 of each experimenter during walking through a millimeter wave radio frequency receiving end 805, finally collecting 9831 walking videos, 3277 Wi-Fi signals and 3277 millimeter wave signals, wherein each walking video and each wireless signal comprise a plurality of Gait cycles (Gait Cycle, which means a walking process from heel to ground again) of the same foot. Then, each walking video is processed to obtain a gait energy diagram (as shown at 901 in fig. 9), each Wi-Fi signal is processed to obtain a Wi-Fi spectrum (as shown at 902 in fig. 9), and each millimeter wave signal is processed to obtain a millimeter wave spectrum (as shown at 903 in fig. 9). Subsequently, gait is utilized And respectively generating a first training sample set, a second training sample set and a third training sample set by the energy map set, the Wi-Fi frequency spectrum set and the millimeter wave frequency spectrum set. Next, the first, second, and third sub-recognition models are trained, respectively, and weight ratios (e.g., 3:1:1) of model outputs of the first, second, and third sub-recognition models in the output of the gait recognition models are determined. Finally, a gait recognition model is obtained, i.e. a combination of the first sub-recognition model, the second sub-recognition model and the third sub-recognition model, which determine the weight ratio.

With further reference to fig. 10, as an implementation of the method shown in fig. 2, the present application provides an embodiment of an apparatus for generating a gait recognition model, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus is particularly applicable to a server.

As shown in fig. 10, the apparatus 1000 for generating a gait recognition model of the present embodiment may include a sample acquisition unit 1001 and a model training unit 1002. Wherein the sample acquisition unit 1001 is configured to acquire a set of training samples, wherein the training samples comprise a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum being wireless spectrums that characterize gait of the same user, the first wireless spectrum and the third wireless spectrum being wireless spectrums that characterize gait of different users; and the model training unit 1002 is configured to train to obtain a gait recognition model based on the training sample set and a preset loss function.

In this embodiment, the specific implementation of the sample acquiring unit 1001 and the model training unit 1002 of the apparatus 1000 for generating a gait recognition model may refer to the related descriptions of steps 201 to 202 in the corresponding embodiment of fig. 2, and will not be repeated here.

In some optional implementations of the present embodiment, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum may be selected from a set of pre-stored wireless spectrums. The radio spectrum in the set of radio spectrums is generated by the steps of: during the period that the sample user walks along the preset route, acquiring a wireless signal from a target position of an area where the sample user is located, and acquiring a wireless signal described by adopting the CSI; denoising the acquired wireless signals; and performing time-frequency conversion on the denoised wireless signals to obtain a wireless spectrum representing gait of the sample user.

In some alternative implementations of the present embodiment, the wireless signals may include Wi-Fi signals or millimeter wave signals.

In some alternative implementations of the present embodiment, the model training unit 1002 may include a training module. Wherein the training module may be configured to perform the following training steps: for training samples in the training sample set, respectively inputting a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum of the training samples into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first wireless frequency spectrum, second wireless frequency spectrum and third wireless frequency spectrum; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained first characteristic information set, second characteristic information set and third characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some optional implementations of this embodiment, the model training unit 1002 may further include an adjustment module. Wherein the adjustment module may be configured to: if the value of the loss function is larger than the preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the training step.

The device for generating the gait recognition model provided by the embodiment of the application is capable of improving the accuracy of gait recognition by acquiring the training sample set including the training samples generated by the wireless spectrum representing the gait of the same user and the wireless spectrum representing the gait of different users, and then training the gait recognition model based on the training sample set and the preset loss function.

With further reference to fig. 11, as an implementation of the method shown in fig. 3, the present application provides an embodiment of an apparatus for recognizing gait, which corresponds to the method embodiment shown in fig. 3, and which is particularly applicable in a server.

As shown in fig. 11, the apparatus 1100 for recognizing gait of the present embodiment may include an information acquisition unit 1101, a recognition unit 1102, a matching unit 1103, and a result generation unit 1104. Wherein the information acquisition unit 1101 is configured to acquire a wireless spectrum characterizing a gait of the user to be identified; the recognition unit 1102 is configured to input a wireless spectrum to the gait recognition model to obtain feature information of a user to be recognized; the matching unit 1103 is configured to match the feature information with pre-stored feature information; and the result generation unit 1104 is configured to generate the recognition result of the user to be recognized based on the matching result.

In this embodiment, specific implementations of the information acquisition unit 1101, the identification unit 1102, the matching unit 1103, and the result generation unit 1104 of the apparatus 1100 for identifying gait may refer to the relevant descriptions of steps 301 to 304 in the corresponding embodiment of fig. 3, and are not repeated herein.

According to the device for identifying gait, the acquired wireless spectrum for representing the gait of the user to be identified is input into the trained gait identification model, then the characteristic information output by the model is matched with the pre-stored characteristic information, and finally the identification result of the user to be identified is generated based on the matching result, so that the accuracy of gait identification can be improved.

With further reference to fig. 12, as an implementation of the method shown in fig. 4, the present application provides another embodiment of an apparatus for generating a gait recognition model, where the apparatus embodiment corresponds to the method embodiment shown in fig. 4, and the apparatus is particularly applicable to a server.

As shown in fig. 12, the apparatus 1200 for generating a gait recognition model of the present embodiment may include a first sample acquisition unit 1201, a second sample acquisition unit 1202, a first model training unit 1203, a second model training unit 1204, and a weight determination unit 1205. Wherein the first sample acquisition unit 1201 is configured to acquire a first set of training samples, wherein the first training sample comprises a first gait energy diagram, a second gait energy diagram and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram being gait energy diagrams representing the gait of the same user, the first gait energy diagram and the third gait energy diagram being gait energy diagrams representing the gait of different users; the second sample acquisition unit 1202 is configured to acquire a second set of training samples, wherein the second training samples comprise a first wireless spectrum, a second wireless spectrum and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum being wireless spectrums that characterize gait of the same user, the first wireless spectrum and the third wireless spectrum being wireless spectrums that characterize gait of different users; the first model training unit 1203 is configured to train to obtain a first sub-recognition model based on the first training sample set and a preset loss function; the second model training unit 1204 is configured to train to obtain a second sub-recognition model based on the second set of training samples and the loss function; and the weight determining unit 1205 is configured to determine weights in the output of the gait recognition model of the model output of the first sub-recognition model and the model output of the second sub-recognition model, respectively, according to the number of the first training samples in the first training sample set and the number of the second training samples in the second training sample set.

In this embodiment, specific implementations of the first sample acquiring unit 1201, the second sample acquiring unit 1202, the first model training unit 1203, the second model training unit 1204 and the weight determining unit 1205 of the apparatus 1200 for generating a gait recognition model may refer to the relevant descriptions about steps 401 to 405 in the corresponding embodiment of fig. 4, and are not repeated herein.

In some alternative implementations of the present embodiments, the first gait energy pattern, the second gait energy pattern and the third gait energy pattern are selected from a pre-stored set of gait energy patterns. The gait energy patterns in the set of gait energy patterns may be generated by: during the walking period of the sample user along the preset route, video acquisition is carried out on the sample user, so that a walking video of the sample user is obtained; extracting a plurality of key frames from the walking video; extracting human body contours of sample users from a plurality of key frames to generate human body silhouettes; human body silhouette based on generation a gait energy diagram is synthesized.

In some optional implementations of the present embodiment, the first wireless spectrum, the second wireless spectrum, and the third wireless spectrum are selected from a set of pre-stored wireless spectrums. The radio spectrum in the set of radio spectrums may be generated by the steps of: during the period that the sample user walks along the preset route, acquiring a wireless signal from a target position of an area where the sample user is located, and acquiring a wireless signal described by adopting the CSI; denoising the acquired wireless signals; and performing time-frequency conversion on the denoised wireless signals to obtain a wireless spectrum representing gait of the sample user.

In some optional implementations of this embodiment, the first model training unit 1203 described above may include a first training module. Wherein, the first training module may be configured to perform a first training step of: for a first training sample in a first training sample set, respectively inputting a first gait energy diagram, a second gait energy diagram and a third gait energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first gait energy diagram, second gait energy diagram and third gait energy diagram; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained first characteristic information set, second characteristic information set and third characteristic information set; and if the value of the loss function is smaller than or equal to a preset numerical value, determining the initial convolutional neural network as a first sub-recognition model.

In some optional implementations of this embodiment, the first model training unit 1203 may further include a first adjustment module. Wherein the first adjustment module is configured to: if the value of the loss function is larger than a preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the first training step.

In some optional implementations of this embodiment, the second model training unit 1204 may include a second training module. Wherein the second training module is configured to perform a second training step as follows: for a second training sample in the second training sample set, respectively inputting a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum of the second training sample into an initial convolutional neural network to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless frequency spectrum, second wireless frequency spectrum and third wireless frequency spectrum; determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained fourth characteristic information set, fifth characteristic information set and sixth characteristic information set; and if the value of the loss function is smaller than or equal to a preset value, determining the initial convolutional neural network as a gait recognition model.

In some optional implementations of this embodiment, the second model training unit 1204 may further include a second adjustment module. Wherein the second adjustment module is configured to: if the value of the loss function is larger than the preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the second training step.

The device for generating the gait recognition model provided by the embodiment of the application can make up for application scenes (for example, different walking routes, different wearing and the like) with low recognition rate of the gait recognition model trained by using a wireless frequency spectrum, so that the accuracy rate of gait recognition is further improved.

With further reference to fig. 13, as an implementation of the method shown in fig. 5, the present application provides another embodiment of an apparatus for recognizing gait, which corresponds to the method embodiment shown in fig. 5, and which is particularly applicable in a server.

As shown in fig. 13, the apparatus 1300 for recognizing gait of the present embodiment may include an information acquisition unit 1301, a first recognition unit 1302, a second recognition unit 1303, a first matching unit 1304, a second matching unit 1305, and a result generation unit 1306. Wherein the information acquisition unit 1301 is configured to acquire a gait energy diagram and a wireless spectrum characterizing the gait of the user to be identified; the first recognition unit 1302 is configured to input a gait energy diagram to a first sub-recognition model of the gait recognition model to obtain first feature information of the user to be recognized; the second identifying unit 1303 is configured to input a wireless spectrum to a second sub-identifying model of the gait identifying model to obtain second characteristic information of the user to be identified; the first matching unit 1304 is configured to match the first characteristic information with pre-stored first characteristic information; the second matching unit 1305 is configured to match the second characteristic information with pre-stored second characteristic information; and the result generation unit 1306 is configured to fuse the matching result of the first characteristic information and the matching result of the second characteristic information based on the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generate a recognition result of the user to be recognized.

In this embodiment, specific implementations of the information acquisition unit 1301, the first identification unit 1302, the second identification unit 1303, the first matching unit 1304, the second matching unit 1305, and the result generation unit 1306 of the apparatus 1300 for identifying gait can refer to the related descriptions about the steps 501 to 506 in the corresponding embodiment of fig. 5, which are not repeated here.

The device for recognizing gait provided in the above embodiment of the present application can make up for application scenarios (for example, different walking routes, different wearing, etc.) in which the recognition rate of the gait recognition model trained by using the wireless spectrum is not high, thereby further improving the accuracy rate of gait recognition.

With further reference to fig. 14, as an implementation of the method shown in fig. 6, the present application provides a further embodiment of an apparatus for generating a gait recognition model, which corresponds to the method embodiment shown in fig. 6, and which is particularly applicable in a server.

As shown in fig. 14, the apparatus 1400 for generating a gait recognition model of the present embodiment may include a first sample acquisition unit 1401, a second sample acquisition unit 1402, a third sample acquisition unit 1403, a first model training unit 1404, a second model training unit 1405, a third model training unit 1406, and a weight determination unit 1407. Wherein the first sample acquisition unit 1401 is configured to acquire a first training sample set, wherein the first training sample comprises a first gait energy diagram, a second gait energy diagram and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram being gait energy diagrams representing the gait of the same user, the first gait energy diagram and the third gait energy diagram being gait energy diagrams representing the gait of different users; the second sample acquisition unit 1402 is configured to acquire a second set of training samples, where the second training sample includes a first Wi-Fi spectrum, a second Wi-Fi spectrum, and a third Wi-Fi spectrum, the first Wi-Fi spectrum and the second Wi-Fi spectrum being Wi-Fi spectrums that characterize gait of the same user, the first Wi-Fi spectrum and the third Wi-Fi spectrum being Wi-Fi spectrums that characterize gait of different users; the third sample acquiring unit 1403 is configured to acquire a third training sample set, where the third training sample includes a first millimeter wave spectrum, a second millimeter wave spectrum, and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum being millimeter wave spectrums representing gait of the same user, the first millimeter wave spectrum and the third millimeter wave spectrum being millimeter wave spectrums representing gait of different users; the first model training unit 1404 is configured to train to obtain a first sub-recognition model based on the first training sample set and a preset loss function; the second model training unit 1405 is configured to train to obtain a second sub-recognition model based on the second set of training samples and the loss function; the third model training unit 1406 is configured to train to obtain a third sub-recognition model based on the third set of training samples and the loss function; the weight determination unit 1407 is configured to determine a weight of the model output of the first sub-recognition model, the model output of the second sub-recognition model, and the model output of the third sub-recognition model in the output of the gait recognition model based on the number of the first training samples in the first training sample set, the number of the second training samples in the second training sample set, and the number of the third training samples in the third training sample set.

In this embodiment, the specific implementation of the first sample acquiring unit 1401, the second sample acquiring unit 1402, the third sample acquiring unit 1403, the first model training unit 1404, the second model training unit 1405, the third model training unit 1406 and the weight determining unit 1407 of the apparatus 1400 for generating a gait recognition model may refer to the related descriptions about steps 601 to 607 in the corresponding embodiment of fig. 6, and are not repeated herein.

The device for generating the gait recognition model provided by the embodiment of the application can make up for the application scene (for example, different walking routes, different wearing and the like) with low recognition rate of the gait recognition model trained by using a single wireless frequency spectrum, thereby further improving the accuracy rate of gait recognition.

With further reference to fig. 15, as an implementation of the method shown in fig. 7, the present application provides a further embodiment of an apparatus for recognizing gait, which corresponds to the method embodiment shown in fig. 7, and which is particularly applicable in a server.

As shown in fig. 15, the apparatus 1500 for recognizing gait of the present embodiment may include an information acquisition unit 1501, a first recognition unit 1502, a second recognition unit 1503, a third recognition unit 1504, a first matching unit 1505, a second matching unit 1506, a third matching unit 1507, and a result generation unit 1508. Wherein the information acquisition unit 1501 is configured to acquire a gait energy diagram, wi-Fi spectrum and millimeter wave spectrum characterizing gait of a user to be identified; the first recognition unit 1502 is configured to input a gait energy pattern to a first sub-recognition model of the gait recognition model to obtain first characteristic information of a user to be recognized; the second recognition unit 1503 is configured to input the Wi-Fi spectrum to a second sub-recognition model of the gait recognition model to obtain second feature information of the user to be recognized; the third recognition unit 1504 is configured to input the millimeter wave spectrum to a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized; the first matching unit 1505 is configured to match the first characteristic information with pre-stored first characteristic information; the second matching unit 1506 is configured to match the second characteristic information with pre-stored second characteristic information; the third matching unit 1507 is configured to match the third characteristic information with pre-stored third characteristic information; the result generating unit 1508 is configured to fuse a matching result of the first feature information, a matching result of the second feature information, and a matching result of the third feature information based on weights in the output of the gait recognition model of the model output of the first sub-recognition model, the model output of the second sub-recognition model, and the model output of the third sub-recognition model, and generate a recognition result of the user to be recognized.

In this embodiment, the specific implementation of the information acquisition unit 1501, the first identification unit 1502, the second identification unit 1503, the third identification unit 1504, the first matching unit 1505, the second matching unit 1506, the third matching unit 1507 and the result generation unit 1508 of the apparatus 1500 for identifying gait may refer to the relevant descriptions of steps 701 to 708 in the corresponding embodiment of fig. 7, and will not be repeated here.

The device for recognizing gait provided in the above embodiment of the present application can make up for application scenarios (for example, different walking routes, different wearing, etc.) in which the recognition rate of the gait recognition model trained by using a single wireless spectrum is not high, thereby further improving the accuracy rate of gait recognition.

Referring now to FIG. 16, there is illustrated a schematic diagram of a computer system 1600 suitable for use in implementing an electronic device (e.g., server 106 of FIG. 1) of an embodiment of the present application. The electronic device shown in fig. 16 is only an example, and should not impose any limitation on the functions and scope of use of the embodiments of the present application.

As shown in fig. 16, the computer system 1600 includes one or more Central Processing Units (CPUs) 1601 that can perform various appropriate actions and processes according to programs stored in a Read Only Memory (ROM) 1602 or programs loaded from a storage portion 1608 into a Random Access Memory (RAM) 1603. In RAM 1603, various programs and data required for the operation of system 1600 are also stored. The CPU 1601, ROM 1602, and RAM 1603 are connected to each other by a bus 1604. An input/output (I/O) interface 1605 is also connected to the bus 1604.

The following components are connected to the I/O interface 1605: an input portion 1606 including a keyboard, a mouse, and the like; an output portion 1607 including, for example, an Organic Light Emitting Diode (OLED) display, a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage section 1608 including a hard disk or the like; and a communication section 1609 including a network interface card such as a LAN card, a modem, or the like. The communication section 1609 performs communication processing via a network such as the internet. The drive 1610 is also connected to the I/O interface 1605 as needed. A removable medium 1611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed as needed on the drive 1610 so that a computer program read out therefrom is installed into the storage section 1608 as needed.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such embodiments, the computer program may be downloaded and installed from a network via the communication portion 1609, and/or installed from the removable media 1611. The above-described functions defined in the method of the present application are performed when the computer program is executed by a Central Processing Unit (CPU) 1601.

It should be noted that, the computer readable medium described in the present application may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM) optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the preceding. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present application, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present application may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units involved in the embodiments of the present application may be implemented by software, or may be implemented by hardware. The described units may also be provided in a processor, for example, described as: a processor includes a sample acquisition unit and a model training unit. The names of these units do not in any way constitute a limitation of the unit itself, for example, the sample acquisition unit may also be described as "unit acquiring a training sample set".

As another aspect, the present application also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by the apparatus, cause the apparatus to: acquiring a training sample set, wherein the training sample comprises a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum, the first wireless frequency spectrum and the second wireless frequency spectrum are wireless frequency spectrums for representing gait of the same user, and the first wireless frequency spectrum and the third wireless frequency spectrum are wireless frequency spectrums for representing gait of different users; and training based on the training sample set and a preset loss function to obtain a gait recognition model.

The foregoing description is only of the preferred embodiments of the present application and is presented as a description of the principles of the technology being utilized. It will be appreciated by persons skilled in the art that the scope of the invention referred to in this application is not limited to the specific combinations of features described above, but it is intended to cover other embodiments in which any combination of features described above or equivalents thereof is possible without departing from the spirit of the invention. Such as the above-described features and technical features having similar functions (but not limited to) disclosed in the present application are replaced with each other.

Claims

1. A method for generating a gait recognition model, the gait recognition model comprising a first sub-recognition model and a second sub-recognition model, the method comprising:

acquiring a first training sample set, wherein the first training sample comprises a first gait energy diagram, a second gait energy diagram and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram are gait energy diagrams for representing the gait of the same user, and the first gait energy diagram and the third gait energy diagram are gait energy diagrams for representing the gait of different users;

acquiring a second training sample set, wherein the second training sample set comprises a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum, the first wireless frequency spectrum and the second wireless frequency spectrum are wireless frequency spectrums for representing the gait of the same user, and the first wireless frequency spectrum and the third wireless frequency spectrum are wireless frequency spectrums for representing the gait of different users;

Training to obtain the first sub-recognition model based on the first training sample set and a preset loss function, wherein the loss function is a function for representing the difference degree of a second difference and a first difference, the first difference represents the difference of characteristic information extracted from training samples of the gait of the same user, and the second difference represents the difference of characteristic information extracted from training samples of the gait of different users;

training to obtain the second sub-recognition model based on the second training sample set and the loss function;

determining weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model respectively according to the number of first training samples in the first training sample set and the number of second training samples in the second training sample set;

wherein the first, second and third gait energy patterns are selected from a pre-stored set of gait energy patterns, the gait energy patterns in the set of gait energy patterns being generated by:

during the walking period of the sample user along the preset route, video acquisition is carried out on the sample user, so that a walking video of the sample user is obtained;

Extracting a plurality of key frames from the walking video;

extracting human body contours of sample users from the plurality of key frames by using a human body segmentation method to generate human body silhouettes;

synthesizing a gait energy diagram based on the generated human body silhouette, wherein the gait energy diagram is defined as:wherein x and y are pixel coordinates, M is the total number of key frames, t is the number of the key frames, G (x, y) represents a gait energy diagram, I (x, y, t) is the pixel value of the pixel with coordinates (x, y) in the key frame numbered t.

2. The method of claim 1, wherein the training to obtain the first sub-recognition model based on the first training sample set and a preset loss function comprises:

the following first training step is performed:

for a first training sample in the first training sample set, respectively inputting a first gait energy diagram, a second gait energy diagram and a third gait energy diagram of the first training sample into an initial convolutional neural network to obtain first characteristic information, second characteristic information and third characteristic information which respectively correspond to the input first gait energy diagram, second gait energy diagram and third gait energy diagram;

determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained first characteristic information set, second characteristic information set and third characteristic information set;

And if the value of the loss function is smaller than or equal to the preset value, determining an initial convolutional neural network as the first sub-recognition model.

3. The method of claim 2, wherein the training to obtain the first sub-recognition model based on the first training sample set and a preset loss function further comprises:

and if the value of the loss function is larger than the preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the first training step.

4. The method of claim 1, wherein the training to obtain the second sub-recognition model based on the second set of training samples and the loss function comprises:

the following second training step is performed:

for a second training sample in the second training sample set, respectively inputting a first wireless frequency spectrum, a second wireless frequency spectrum and a third wireless frequency spectrum of the second training sample into an initial convolutional neural network to obtain fourth characteristic information, fifth characteristic information and sixth characteristic information which respectively correspond to the input first wireless frequency spectrum, second wireless frequency spectrum and third wireless frequency spectrum;

Determining whether the value of the loss function is smaller than or equal to a preset numerical value based on the obtained fourth characteristic information set, fifth characteristic information set and sixth characteristic information set;

and if the value of the loss function is smaller than or equal to the preset value, determining an initial convolutional neural network as the gait recognition model.

5. The method of claim 4, wherein the training to obtain the second sub-recognition model based on the second set of training samples and the loss function further comprises:

and if the value of the loss function is larger than the preset value, adjusting parameters of the initial convolutional neural network, using the adjusted initial convolutional neural network as the initial convolutional neural network, and continuing to execute the second training step.

6. The method of claim 1, wherein the first, second, and third wireless spectrum are selected from a pre-stored set of wireless spectrums, the wireless spectrums of the set of wireless spectrums being generated by the steps of:

during the period that the sample user walks along the preset route, acquiring a wireless signal from a target position of an area where the sample user is located, and acquiring a wireless signal described by adopting the CSI;

Denoising the acquired wireless signals;

and performing time-frequency conversion on the denoised wireless signals to obtain a wireless spectrum representing gait of the sample user.

7. The method of claim 6, wherein the wireless signal comprises a Wi-Fi signal or a millimeter wave signal.

8. A method for recognizing a gait, comprising the following steps:

acquiring a gait energy diagram and a wireless frequency spectrum for representing the gait of a user to be identified;

inputting the gait energy diagram into a first sub-recognition model of a gait recognition model generated by the method of one of claims 1 to 7 to obtain first characteristic information of the user to be recognized;

inputting the wireless spectrum to a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized;

matching the first characteristic information with pre-stored first characteristic information;

matching the second characteristic information with pre-stored second characteristic information;

and fusing the matching result of the first characteristic information and the matching result of the second characteristic information based on the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, so as to generate the recognition result of the user to be recognized.

9. A method for generating a gait recognition model, the gait recognition model comprising a first sub-recognition model, a second sub-recognition model and a third sub-recognition model, the method comprising:

acquiring a second training sample set, wherein the second training sample set comprises a first Wi-Fi frequency spectrum, a second Wi-Fi frequency spectrum and a third Wi-Fi frequency spectrum, the first Wi-Fi frequency spectrum and the second Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing gait of the same user, and the first Wi-Fi frequency spectrum and the third Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing gait of different users;

acquiring a third training sample set, wherein the third training sample set comprises a first millimeter wave spectrum, a second millimeter wave spectrum and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectrums for representing gait of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectrums for representing gait of different users;

Training to obtain the first sub-recognition model based on the first training sample set and a preset loss function, wherein the loss function is a function for representing the difference degree of the second difference and the first difference, the first difference characterizes the difference of the characteristic information extracted from the training samples of the gait of the same user, and the second difference characterizes the difference of the characteristic information extracted from the training samples of the gait of different users;

training to obtain the third sub-recognition model based on the third training sample set and the loss function;

determining weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model according to the number of first training samples in the first training sample set, the number of second training samples in the second training sample set and the number of third training samples in the third training sample set;

extracting a plurality of key frames from the walking video;

synthesizing a gait energy diagram based on the generated human body silhouette, wherein the gait energy diagram is defined as:where x and y are pixel coordinates, M is the total number of key frames, t is the number of key frames, G (x, y) represents the gait energy diagram, and I (x, y, t) is the pixel value of the pixel with coordinates (x, y) in the key frame with number t.

10. A method for identifying gait, comprising:

acquiring a gait energy diagram, wi-Fi frequency spectrum and millimeter wave frequency spectrum which characterize gait of a user to be identified;

inputting the gait energy diagram into a first sub-recognition model of a gait recognition model generated by the method of claim 9 to obtain first characteristic information of the user to be recognized;

inputting the Wi-Fi frequency spectrum to a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized;

inputting the millimeter wave spectrum to a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized;

matching the third characteristic information with pre-stored third characteristic information;

and fusing the matching result of the first characteristic information, the matching result of the second characteristic information and the matching result of the third characteristic information based on the weights of the model output of the first sub-recognition model, the model output of the second sub-recognition model and the model output of the third sub-recognition model in the output of the gait recognition model, so as to generate the recognition result of the user to be recognized.

11. An apparatus for generating a gait recognition model, the gait recognition model comprising a first sub-recognition model and a second sub-recognition model, the apparatus comprising:

a first sample acquisition unit configured to acquire a first training sample set, wherein the first training sample includes a first gait energy diagram, a second gait energy diagram, and a third gait energy diagram, the first gait energy diagram and the second gait energy diagram being gait energy diagrams that characterize the gait of the same user, the first gait energy diagram and the third gait energy diagram being gait energy diagrams that characterize the gait of different users;

A second sample acquisition unit configured to acquire a second set of training samples, wherein the second training samples include a first wireless spectrum, a second wireless spectrum, and a third wireless spectrum, the first wireless spectrum and the second wireless spectrum being wireless spectrums that characterize gait of the same user, the first wireless spectrum and the third wireless spectrum being wireless spectrums that characterize gait of different users;

a first model training unit configured to train to obtain the first sub-recognition model based on the first training sample set and a preset loss function, wherein the loss function is a function for representing a difference degree of a second difference and a first difference, the first difference represents a difference of characteristic information extracted from training samples of gait of the same user, and the second difference represents a difference of characteristic information extracted from training samples of gait of different users;

a second model training unit configured to train to obtain the second sub-recognition model based on the second training sample set and the loss function;

a weight determining unit configured to determine weights of model outputs of the first sub-recognition model and the second sub-recognition model in the output of the gait recognition model, respectively, according to the number of first training samples in the first training sample set and the number of second training samples in the second training sample set;

extracting a plurality of key frames from the walking video;

synthesizing a gait energy diagram based on the generated human body silhouette, wherein the gait energy diagram is defined as:wherein x and y are pixel coordinates, M is the total number of key frames, t is the number of key frames, G (x, y) represents gait energy diagram, I (x, y, t) is the switch with the number tThe coordinates in the key frame are (x, y) pixel values of the pixels.

12. The apparatus of claim 11, wherein the first model training unit comprises:

a first training module configured to perform a first training step of:

if the value of the loss function is less than or equal to the preset value, an initial convolutional neural network is determined as the first sub-recognition model.

13. The apparatus of claim 12, wherein the first model training unit further comprises:

and the first adjusting module is configured to adjust parameters of the initial convolutional neural network if the value of the loss function is larger than the preset value, and continue to execute the first training step by using the adjusted initial convolutional neural network as the initial convolutional neural network.

14. The apparatus of claim 11, wherein the second model training unit comprises:

a second training module configured to perform a second training step of:

15. The apparatus of claim 14, wherein the second model training unit further comprises:

and the second adjusting module is configured to adjust parameters of the initial convolutional neural network if the value of the loss function is larger than the preset value, and continue to execute the second training step by using the adjusted initial convolutional neural network as the initial convolutional neural network.

16. The apparatus of claim 11, wherein the first, second, and third wireless spectrum are selected from a pre-stored set of wireless spectrums, the wireless spectrums of the set of wireless spectrums being generated by the steps of:

denoising the acquired wireless signals;

17. The apparatus of claim 16, wherein wireless signals comprise Wi-Fi signals or millimeter wave signals.

18. An apparatus for recognizing gait, comprising:

an information acquisition unit configured to acquire a gait energy diagram and a wireless spectrum characterizing a gait of a user to be identified;

a first recognition unit configured to input the gait energy pattern to a first sub-recognition model of a gait recognition model generated by the method of one of claims 1 to 7 to obtain first characteristic information of the user to be recognized;

a second recognition unit configured to input the wireless spectrum to a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized;

a first matching unit configured to match the first characteristic information with pre-stored first characteristic information;

a second matching unit configured to match the second characteristic information with pre-stored second characteristic information;

and the result generating unit is configured to fuse the matching result of the first characteristic information and the matching result of the second characteristic information based on the weights of the model output of the first sub-recognition model and the model output of the second sub-recognition model in the output of the gait recognition model, and generate the recognition result of the user to be recognized.

19. An apparatus for generating a gait recognition model, the gait recognition model comprising a first sub-recognition model, a second sub-recognition model and a third sub-recognition model, the apparatus comprising:

the second sample acquisition unit is configured to acquire a second training sample set, wherein the second training sample comprises a first Wi-Fi frequency spectrum, a second Wi-Fi frequency spectrum and a third Wi-Fi frequency spectrum, the first Wi-Fi frequency spectrum and the second Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing the gait of the same user, and the first Wi-Fi frequency spectrum and the third Wi-Fi frequency spectrum are Wi-Fi frequency spectrums for representing the gait of different users;

a third sample acquisition unit configured to acquire a third training sample set, wherein the third training sample includes a first millimeter wave spectrum, a second millimeter wave spectrum, and a third millimeter wave spectrum, the first millimeter wave spectrum and the second millimeter wave spectrum are millimeter wave spectrums representing gait of the same user, and the first millimeter wave spectrum and the third millimeter wave spectrum are millimeter wave spectrums representing gait of different users;

a third model training unit configured to train to obtain the third sub-recognition model based on the third training sample set and the loss function;

a weight determining unit configured to determine weights in the output of the gait recognition model of the model output of the first sub-recognition model, the model output of the second sub-recognition model, and the model output of the third sub-recognition model according to the number of first training samples in the first training sample set, the number of second training samples in the second training sample set, and the number of third training samples in the third training sample set;

extracting a plurality of key frames from the walking video;

20. An apparatus for recognizing gait, comprising:

an information acquisition unit configured to acquire a gait energy diagram, wi-Fi spectrum, and millimeter wave spectrum that characterize gait of a user to be identified;

a first recognition unit configured to input the gait energy diagram to a first sub-recognition model of a gait recognition model generated by the method of claim 9 to obtain first feature information of the user to be recognized;

The second recognition unit is configured to input the Wi-Fi frequency spectrum into a second sub-recognition model of the gait recognition model to obtain second characteristic information of the user to be recognized;

a third recognition unit configured to input the millimeter wave spectrum to a third sub-recognition model of the gait recognition model to obtain third characteristic information of the user to be recognized;

a third matching unit configured to match the third characteristic information with pre-stored third characteristic information;

and a result generation unit configured to fuse a matching result of the first feature information, a matching result of the second feature information, and a matching result of the third feature information based on weights of a model output of the first sub-recognition model, a model output of the second sub-recognition model, and a model output of the third sub-recognition model in an output of the gait recognition model, and generate a recognition result of the user to be recognized.

21. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-10.

22. A computer readable medium having stored thereon a computer program, wherein the program when executed by a processor implements the method of any of claims 1-10.