KR20230032712A - Apparatus and method for recognizing gait using noise reduction network - Google Patents

Abstract

The present invention relates to a gait recognition technology and more specifically, relates to a device and method for recognizing a gait based on a noise improvement network. According to an embodiment of the present invention, by removing a noise through the noise improvement network, a gait recognition performance can be efficiently improved. The device comprises: a gait information input part; a joint model extraction part; a preprocessing part; a joint model improvement part; a gait pattern extraction part; and a gait recognition part.

Description

Noise enhancement network based gait recognition device and method {APPARATUS AND METHOD FOR RECOGNIZING GAIT USING NOISE REDUCTION NETWORK}

The present invention relates to gait recognition technology, and more particularly, to a gait recognition device and method based on a noise reduction network.

Biometrics technology is a technology that recognizes a person's identity based on physiological characteristics or behavioral characteristics, and has been mainly used in the area of access control in the past. Biometric technology has recently been used in various fields such as smart devices, national defense, healthcare, and fintech. In particular, gait recognition technology among biometric technologies is expected to show a high average annual growth rate as non-face-to-face and non-contact services dominate after the COVID-19 crisis.

Gait recognition technology includes CNN or RNN deep learning technology that takes RGB data and motion flow as inputs. However, since deep learning technology uses the entire area of video frames for learning as input information, it may not be focused on human behavior and may cause performance degradation due to various types of noise.

The background art of the present invention is published in Korean Registered Patent No. 10-2101221.

The present invention provides a gait recognition device and method based on a noise reduction network that automatically generates gait information necessary for gait recognition without a person directly acquiring it by using a non-contact input sensor.

The present invention provides an apparatus and method for gait recognition based on a noise amelioration network capable of efficiently improving gait recognition performance by removing noise through the noise amelioration network.

The technical problem to be achieved by the present invention is not limited to the above-mentioned technical problem, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

According to one aspect of the present invention, a noise reduction network-based gait recognition device is provided.

An apparatus for gait recognition based on a noise-reducing network according to an embodiment of the present invention includes a gait information input unit for inputting gait information of an object generated through a non-contact sensor, and a joint for extracting a joint model from the gait information of an object and generating joint model data. A model extraction unit, a preprocessing unit that preprocesses joint model data, a joint model improvement unit that generates improved joint model data from preprocessed joint model data using a pre-learned noise improvement network, and an improved joint model data and a gait pattern extractor for extracting a gait pattern based on the gait pattern and a gait recognition unit for performing gait recognition by calculating a similarity comparison between the gait pattern and an identity feature.

According to another aspect of the present invention, a noise reduction network-based gait recognition method is provided.

A noise reduction network-based gait recognition method according to an embodiment of the present invention includes the steps of inputting gait information of an object generated through a non-contact sensor, extracting a joint model from the gait information of an object and generating joint model data, Performing preprocessing on model data, generating improved joint model data from preprocessed joint model data using a pre-learned noise improvement network, extracting a gait pattern based on the improved joint model data, and and performing gait recognition by calculating a similarity comparison between the gait pattern and the identity feature.

According to an embodiment of the present invention, gait information necessary for gait recognition can be automatically generated using a non-contact input sensor without a person directly obtaining it.

Also, according to an embodiment of the present invention, gait recognition performance can be efficiently improved by removing noise through a noise improvement network.

The effects of the present invention are not limited to the above effects, and should be understood to include all effects that can be inferred from the description of the present invention or the configuration of the invention described in the claims.

1 and 2 are diagrams for explaining a noise reduction network-based gait recognition device.
3 is a diagram for explaining a noise reduction network-based gait recognition method;
4 and 5 are diagrams illustrating gait recognition performance of the gait recognition device based on a noise reduction network according to an embodiment of the present invention.

Hereinafter, the present invention will be described with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and, therefore, is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected (connected, contacted, combined)" with another part, this is not only "directly connected", but also "indirectly connected" with another member in between. "Including cases where In addition, when a part "includes" a certain component, it means that it may further include other components without excluding other components unless otherwise stated.

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "include" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

1 and 2 are diagrams for explaining a noise reduction network-based gait recognition device.

Referring to FIG. 1, the noise reduction network-based gait recognition device 100 includes a gait information input unit 110, a joint model extraction unit 120, a pre-processing unit 130, a joint model improvement unit 140, and a gait pattern extraction unit. (150) and a gait recognition unit (160).

The gait information input unit 110 may input gait information of an object generated through a non-contact sensor. Here, the non-contact sensor may include at least one of a LiDAR sensor and an RGB sensor. For example, the gait information input unit 110 may generate and input gait information of an object based on shape information or depth information of an object measured by a non-contact sensor.

The joint model extraction unit 120 extracts a joint model from the gait information of the object input from the gait information input unit 110 . Specifically, the joint model extraction unit 120 may extract a joint model by estimating a joint from gait information of an object. At this time, the joint model extraction unit 120 may estimate the joint from the gait information of the object through a convolutional neural network (CNN). The joint model extraction unit 120 according to an embodiment of the present invention estimates a joint using a Transformer network. The joint model extraction unit 120 may extract a joint model of an object based on a joint estimated through a transformer network and generate joint model data.

The preprocessor 130 performs preprocessing on the joint model data generated from the joint model extractor 120 .

For example, the pre-processing unit 130 may change at least one of a resolution and a data type of joint model data. At this time, the pre-processing unit 130 changes the resolution of data from FHD to HD according to the network input format of the convolutional neural network (CNN) used from the joint model extraction unit 120, or changes the data format to W * H * 3 can be changed from to B*3*W*H.

In addition, the pre-processing unit 130 may generate a repetitive image for recognizing minimum unit actions in the joint model data. Here, the minimum unit action means an action with a fairly short duration of action, such as raising an arm, walking one step, or sitting down. The pre-processing unit 130 may generate a repetitive image obtained by merging joint model data at least twice or more in order to recognize a minimum unit action from gait information of an object having a short action duration.

The joint model improvement unit 140 removes noise from the preprocessed joint model data by using a pre-learned noise improvement network. Here, the noise enhancement network may use a predictively encoded graph convolutional network (PeGCN), which is a predictively encoded graph convolutional network. Specifically, PeGCN is a network that performs noise improvement by numerically measuring the dependence between input original data and preprocessed data using mutual information. Noise improvement is performed by maximizing the common denominator of the mapping results into latent space.

At this time, mutual information capable of measuring mutual dependence between the two data can be defined as in Equation 1 below.

[Equation 1]

)를 최대화하여 노이즈에 강인한 기본 잠재 변수를 추출한다.Here, x and α are the expression values of the original data and the noise data, respectively, and I(x; α is the mutual information between the two expression values.

) to extract basic latent variables that are robust to noise.

Referring to FIG. 2, PeGCN maximizes the common denominator between the two data by inputting original data X (normal skeleton X) and preprocessed joint model image, noise data X' (noisy skeleton X') during learning.

와 자기회귀(autoregressive) 모듈

로 구성되며,

는 입력되는 데이터를 잠재 공간(Latent space)으로 인코딩하고,

는 잠재 특징(Latent feature)인 α, α'을 추출한다.Specifically, PeGCN is a graph convolutional network (GCN) module

and the autoregressive module

consists of,

encodes the input data into a latent space,

extracts α and α′, which are latent features.

와 자기회귀(autoregressive) 모듈

은 모두 손실을 최적화하고 상호 정보를 최대화하도록 학습하며, 이 때 손실함수(

)는 아래 수학식 2와 같이 나타낼 수 있다. At this time, the graph convolution network (GCN) module

and the autoregressive module

both learn to optimize loss and maximize mutual information, at which time the loss function (

) can be expressed as in Equation 2 below.

[Equation 2]

)를 최대화하지만, 보행 인식에 있어 보행 동작 클래스 식별을 수행해야 하므로 아래의 수학식 3과 같이 교차 엔트로피 손실 함수(

)를 이용한다. PeGCN minimizes the loss so that the mutual information I(x; α

) is maximized, but since gait motion class identification must be performed in gait recognition, the cross entropy loss function (as shown in Equation 3 below)

) is used.

[Equation 3]

)는 교차 엔트로피 손실 함수(

)와 균형 가중치(λ를 사용한 손실 함수(

)의 합으로 나타낸다. Here, C is the number of walking motion classes. o denotes the output of the fully connected network for the classification task in the inference step. That is, the total loss function for learning PeGCN (

) is the cross entropy loss function (

) and the loss function using the balance weights (λ) (

) as the sum of

The joint model improvement unit 140 takes the preprocessed joint model data as an input and generates improved joint model data from which noise is removed through pre-learned PeGCN.

Referring back to FIG. 1 , the gait pattern extraction unit 150 extracts a gait pattern based on the improved joint model data. Specifically, the gait pattern extractor 150 tracks the direction and speed of the position of the joint from the improved joint model data to estimate the joint feature, which is the movement of the body, and estimates the external feature of the object from the RGB image. The gait pattern extraction unit 150 finally extracts a complex gait pattern by fusing the estimated joint characteristics and external appearance characteristics.

At this time, the gait pattern extractor 150 selects only the joints corresponding to each object from the improved joint model data when a plurality of objects included in the improved joint model data and different objects are overlapped (Occluded), and After segmentation, the gait pattern can be extracted by estimating and converging joint features and external features.

The gait recognition unit 160 identifies an individual's identity by calculating a cosine similarity between the gait pattern and the identity characteristics of the individual registered in advance. The gait recognition unit 160 calculates the similarity between the extracted gait pattern and the previously registered individual gait pattern, and selects the highest similarity. At this time, the gait recognition unit 160 may output an identity corresponding to the highest degree of similarity if the selected similarity is greater than a preset threshold, and may classify the identity as unknown if it is less than the preset threshold.

3 is a diagram for explaining a noise reduction network-based gait recognition method.

Referring to FIG. 3 , in step S310, the noise reduction network-based gait recognition apparatus 100 inputs gait information of an object generated through a non-contact sensor. Here, the non-contact sensor may include at least one of a LiDAR sensor and an RGB sensor.

The noise reduction network-based gait recognition apparatus 100 may generate and input gait information of an object based on shape information or depth information of the object measured by the non-contact sensor.

In step S320, the noise reduction network-based gait recognition apparatus 100 extracts a joint model from the gait information of the object input from the gait information input unit 110. Specifically, the noise-reducing network-based gait recognition apparatus 100 may extract a joint model by estimating a joint from gait information of an object. The noise amelioration network-based gait recognition apparatus 100 estimates joints from gait information of an object through a convolutional neural network (CNN), and at this time, the joints are estimated using a transformer network. The noise amelioration network-based gait recognition apparatus 100 may extract a joint model of an object based on a joint estimated through a transformer network and generate joint model data.

In step S330, the noise-reducing network-based gait recognition apparatus 100 performs pre-processing on the joint model data generated from the joint model extractor 120.

Here, the noise reduction network-based gait recognition apparatus 100 may change at least one of a resolution and a data format of the joint model data.

In addition, the noise-reducing network-based gait recognition apparatus 100 may generate a repetitive image obtained by merging joint model data at least twice or more in order to recognize a minimum unit action from gait information of an object having a short action duration.

In step S340, the noise reduction network-based gait recognition apparatus 100 removes noise from the preprocessed joint model data by using the pre-learned noise reduction network. Here, the noise enhancement network may use a predictively encoded graph convolutional network (PeGCN), which is a predictively encoded graph convolutional network.

The noise-reducing network-based gait recognition apparatus 100 takes preprocessed joint model data as an input and generates improved joint model data from which noise is removed through pre-learned PeGCN.

In step S350, the noise reduction network-based gait recognition apparatus 100 extracts a gait pattern based on the improved joint model data. Specifically, the noise-reducing network-based gait recognition apparatus 100 tracks the direction and speed of the position change of the joint from the improved joint model data to estimate the joint feature, which is the movement of the body, and estimates the external feature of the object from the RGB image. do. The noise-reducing network-based gait recognition apparatus 100 finally extracts a complex gait pattern by converging each estimated joint feature and external feature.

At this time, the noise-reducing network-based gait recognition apparatus 100 detects only joints corresponding to each object in the improved joint model data when a plurality of objects included in the improved joint model data and different objects overlap each other. After selecting and dividing the region, the gait pattern can be extracted by estimating and converging joint and external features.

At this time, when there are a plurality of objects included in the improved joint model data, the noise reduction network-based gait recognition apparatus 100 divides a region corresponding to the joint model of each object in the improved joint model data, and then determines the joint characteristics and Gait patterns can be extracted by extracting and converging external features.

In step S360, the noise-reducing network-based gait recognition apparatus 100 calculates a cosine similarity between the gait pattern and the previously registered identity characteristics of the individual to identify the individual's identity. The noise-reducing network-based gait recognition apparatus 100 calculates the similarity between the extracted gait pattern and the previously registered individual gait pattern, and selects the highest similarity. At this time, the noise amelioration network-based gait recognition apparatus 100 may output an identity corresponding to the highest similarity if the selected similarity is greater than a preset threshold, and may classify the identity as unknown if the selected similarity is less than the preset threshold.

4 and 5 are diagrams illustrating gait recognition performance of the gait recognition device based on a noise reduction network according to an embodiment of the present invention.

Referring to FIG. 4 , the noise reduction network-based gait recognition apparatus 100 evaluates gait recognition performance using the NTU-RGB+D data set and the Kinetics-Skeleton data set. Here, the NTU-RGB+D data set is one of the largest data sets in relation to joint model-based gait motions, and includes about 56,000 samples extracted from gait motions in about 60 indoor activities. In addition, the NTU-RGB+D data set was extracted using Microsoft Kinect v2 from three different angles (-45, 0, 45) of the walking motions of 40 volunteers, cross-view and cross-view. It can be divided into cross-subject benchmark protocols. The Kinetics-Skeleton data set is a large-scale gait motion data set generated from the Kinetics data set and contains 34,000 samples.

Referring to (a) to (c) of FIG. 4, walking using NTU-RGB+D cross-view, cross-subject, and Kinetics-Skeleton data sets with noise As a result of motion recognition, unlike other gait recognition methods whose performance rapidly deteriorates as the noise level increases, PeGCN showed higher gait recognition performance as the noise level increased, and higher accuracy than other gait recognition methods at all noise levels. was recorded. Through this, it can be confirmed that the gait recognition performance of the noise-reducing network-based gait recognition apparatus 100 according to an embodiment of the present invention is excellent.

Referring to FIG. 5, as a result of measuring the accuracy of gait motion according to the noise level, the accuracy of other methods except PeGCN is greatly reduced as the noise level increases, whereas the PeGCN is in a noisy environment (i.e., noise level 13). 90% accuracy was maintained. Through this, it can be confirmed that the gait recognition performance of the noise-reducing network-based gait recognition apparatus 100 according to an embodiment of the present invention is excellent.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present invention.

100: Noise improvement network-based gait recognition device
110: gait information input unit
120: joint model extraction unit
130: pre-processing unit
140: joint model improvement unit
150: walking pattern extraction unit
160: gait recognition unit

Claims (11)

In the noise improvement network-based gait recognition device,
A gait information input unit for inputting gait information of an object generated through a non-contact sensor;
a joint model extraction unit for extracting a joint model from gait information of the object and generating joint model data;
a pre-processing unit performing pre-processing on the joint model data;
a joint model improvement unit for generating improved joint model data from the preprocessed joint model data using a pre-learned noise improvement network;
a gait pattern extraction unit for extracting a gait pattern based on the improved joint model data; and
A noise reduction network-based gait recognition device comprising a gait recognition unit configured to identify an identity by calculating a similarity comparison between the gait pattern and an identity feature.
According to claim 1,
The non-contact sensor
A gait recognition device based on a noise amelioration network that is at least one of a LiDAR sensor and an RGB sensor.
According to claim 1,
The pre-processing unit
A noise amelioration network-based gait recognition device that changes at least one of resolution and shape of joint model data.
According to claim 1,
The gait pattern extraction unit
A noise-reducing network-based gait recognition device for estimating joint features and external features from the improved joint model data and extracting a gait pattern by fusion.
According to claim 1,
The gait recognition unit
A noise-reducing network-based gait recognition device for outputting an identity corresponding to the highest similarity when the similarity is greater than a preset threshold and classifying the identity as unknown when the similarity is less than a preset threshold.
In the noise improvement network-based gait recognition method,
inputting gait information of an object generated through a non-contact sensor;
extracting a joint model from gait information of the object and generating joint model data;
performing pre-processing on the joint model data;
generating improved joint model data from the preprocessed joint model data using a pre-learned noise improvement network;
Extracting a gait pattern based on the improved joint model data; and
and identifying an identity by calculating a similarity comparison between the gait pattern and the identity feature.
According to claim 6,
The non-contact sensor
Gait recognition method based on noise amelioration network that is at least one of lidar sensor and RGB sensor.
According to claim 6,
The step of performing preprocessing on the joint model data
A noise enhancement network-based gait recognition method that changes at least one of resolution and shape of joint model data.
The step of extracting a gait pattern based on the improved joint model data
A noise-reducing network-based gait recognition method for estimating joint features and external features from the improved joint model data and extracting a gait pattern by convergence.
The step of identifying the identity is
A noise-reducing network-based gait recognition method for outputting an identity corresponding to the highest similarity when the similarity is greater than a preset threshold and classifying the identity as unknown when the similarity is less than a preset threshold.
A computer program that executes the noise reduction network-based gait recognition method of claim 6 and is recorded on a computer-readable recording medium.

Images