CN114818788A - Tracking target state identification method and device based on millimeter wave perception - Google Patents
Tracking target state identification method and device based on millimeter wave perception Download PDFInfo
- Publication number
- CN114818788A CN114818788A CN202210360722.2A CN202210360722A CN114818788A CN 114818788 A CN114818788 A CN 114818788A CN 202210360722 A CN202210360722 A CN 202210360722A CN 114818788 A CN114818788 A CN 114818788A
- Authority
- CN
- China
- Prior art keywords
- state
- data
- target
- point cloud
- millimeter wave
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
Abstract
The invention provides a method and a device for identifying a tracked target state based on millimeter wave perception, wherein the method comprises the following steps: acquiring point cloud data in real time based on a millimeter wave signal received by a millimeter wave radar; tracking a target based on the obtained point cloud data, and determining space coordinate data of the tracked target; carrying out data splitting on point cloud data which are in a time window with a preset length and are related to a tracking target according to frames, and generating a multi-dimensional data matrix corresponding to each frame of data based on the split data; inputting the multidimensional data matrix into a pre-trained neural network model, wherein the neural network model comprises a plurality of convolutional layers, an LSTM network layer and a full-connection classifier, the full-connection classifier outputs a prediction result of the state of a tracked target, and the state of the tracked target comprises a plurality of action states; and determining the final state of the tracking target based on the tracking target state prediction result and a pre-established finite state machine.
Description
Technical Field
The invention relates to the technical field of millimeter wave sensing, in particular to a method and a device for identifying a target tracking state based on millimeter wave sensing.
Background
Millimeter wave technology can provide sub-millimeter accuracy and can penetrate certain specific materials, such as plastics, clothing, etc., and is not susceptible to environmental conditions such as rain, fog, dust, and snow. Millimeter waves can also be used as a very valuable sensing technology, and can detect targets and provide the distance, the speed and the angle of the targets. Millimeter wave sensors have been widely used in the fields of autopilot, industry, unmanned aerial vehicles, and medical applications. Various millimeter wave perception technologies, such as gesture recognition, gait recognition and heartbeat breath recognition based on millimeter waves, can provide more intelligent, convenient, interesting product experience for the user, and along with the rapid development and maturity of the 5G technology, the millimeter wave wireless module will be widely installed on cell-phones, wearable equipment and more internet of things equipment.
The millimeter wave technology can realize non-contact personnel state identification without a camera, and the millimeter wave personnel state identification device has the advantages of privacy protection, good comfort, passive monitoring and the like. Existing state recognition also includes video-based state recognition technology, wearable sensor-based state recognition technology, and RF signal-based state recognition technology.
The video-based state recognition technology needs to install a camera device in a sensing range, capture continuous motion changes of a target through continuous video shooting, detect the position of the target in each frame by using algorithms such as target detection and the like, extract the motion finished by the target continuously in multiple frames through a state recognition algorithm, and usually adopt a machine learning or deep learning method for state recognition. Because the camera equipment can be used to the state identification technique based on the video, consequently to some sensitive environment, such as environment such as bedroom, bathroom, can have the risk of privacy disclosure, and user's acceptance is lower, and the cost is higher simultaneously. And the video-based method has high requirements on light and cannot work in adverse environments such as smoke, rain, snow and the like.
The state recognition technology based on wearable equipment requires a user to actively wear related sensor equipment, changes of limbs or trunk of the user are captured through instruments such as a gyroscope and an acceleration sensor, sensor signal change curves of the user in different states can be obtained through data denoising, processing and analyzing, and the state of the user is recognized through matching with a state curve template or other recognition modes. However, the wearing of the sensor affects the user experience, the comfort is low, and the continuity of data collection depends on the user's fitness. In addition, additional burdens such as charging of the device, periodic maintenance, and the like exist. And the sensor is worn on different parts of the user, such as wrists or ankles, and the acquired data mainly focuses on the change of the part when a certain action is completed, but not on the whole body movement information of the user, so that the recognizable action is less, the accuracy is not very high, for example, the recognizable rate for slow falling is low, and the like.
The state recognition technology based on RF (radio frequency) signals is to acquire the position and state of an object by using reflected signals by emitting RF waves through radar. However, the distance measured by the conventional RF signal has much lower resolution than the millimeter wave signal, and the positioning and tracking are not as fine as the millimeter wave signal.
At present, a technology for realizing track tracking and fall detection by using Frequency Modulated Continuous Wave (FMCW) as a millimeter wave signal is available, but at present, the technology is limited to recognizing limited states, such as fall, that is, the recognizable state is single, and in addition, the accuracy is not high enough by using a threshold value to perform logic judgment based on point cloud data information.
Although millimeter wave recognition technology is beginning to be applied to recognition of personnel states at present, how to more accurately realize recognition of more kinds of personnel states still remains a problem to be solved.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for identifying a state of a tracking target based on millimeter wave sensing, so as to obviate or mitigate one or more of the disadvantages in the prior art.
One aspect of the present invention provides a method for identifying a target tracking state based on millimeter wave sensing, which comprises the following steps:
acquiring point cloud data in real time based on a millimeter wave signal received by a millimeter wave radar, wherein the point cloud data comprises space coordinate data, Doppler velocity information and signal-to-noise ratio of each detection point;
tracking a target based on the obtained point cloud data, and determining space coordinate data of the tracked target;
performing data splitting on point cloud data which is in a time window with a preset length and is related to the tracking target by frames, and generating a multi-dimensional data matrix corresponding to each frame of data based on the split data, wherein the multi-dimensional data matrix at least comprises data of a plurality of attributes, and the data of the plurality of attributes comprises a spatial coordinate attribute and at least one of the following attributes: doppler velocity attributes and signal-to-noise ratio attributes;
inputting a multi-dimensional data matrix corresponding to each frame data into a pre-trained neural network model, and outputting a tracking target state prediction result, wherein the pre-trained neural network model comprises a multi-layer convolution layer with the channel number being the same as the attribute number of the data in the multi-dimensional data matrix, an LSTM network layer and a full-connection classifier; inputting data of each attribute in a multi-dimensional data matrix corresponding to each frame data into a multi-layer convolution layer of a corresponding channel respectively, outputting a characteristic sequence corresponding to each frame as the input of an LSTM network layer, fusing the output of the LSTM network layer and inputting the fused output into a full-connection classifier, and outputting a prediction result of the state of a tracked target by the full-connection classifier, wherein the state of the tracked target comprises a plurality of action states;
and determining the final state of the tracking target based on the tracking target state prediction result and a pre-established finite state machine.
In some embodiments of the present invention, the feature sequence corresponding to each frame output by the multilayer convolutional layer is a one-dimensional feature sequence, and the fully-connected classifier outputs a prediction result of a state of a tracking target as a prediction probability value of each action state.
In some embodiments of the invention, the action state is selected from some or all of the following states: standing, falling, walking, running, jumping, sitting and lying.
In some embodiments of the invention, the method further comprises: and training a neural network model.
In some embodiments of the present invention, the central coordinate of the multidimensional data matrix is a spatial coordinate of a tracking target, and the generating the multidimensional data matrix corresponding to each frame of data based on the split data includes:
and when the number of the detection points in the point cloud data of the current frame is less than the preset number, filling the blank data by using the part of the detection points of the previous frame closest to the target center point of the current frame until the number of the detection points in the point cloud data of the current frame reaches the preset number.
In some embodiments of the present invention, the pre-established finite state machine contains a transformation relation between states using each action state as a node and a state prediction probability minimum threshold.
In some embodiments of the invention, said determining a final state of the tracking target based on said tracking target state prediction result and a pre-established finite state machine comprises,
the final state of the tracking target is a state which accords with the transformation relation between the states and has a prediction probability value larger than a preset probability threshold value, or is a state which is not changed.
In some embodiments of the invention, the method further comprises: and generating prompt information and/or sending the prompt information to the communication terminal based on the final state of the tracking target determined by the finite state machine.
Another aspect of the present invention provides a tracking target state recognition apparatus based on millimeter wave sensing, which includes a processor and a memory, wherein the memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus implements the steps of the method as described above.
In another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is characterized in that the program, when executed by a processor, implements the steps of the method as described above.
The method and the device for identifying the state of the tracking target based on millimeter wave perception can identify the action state of the tracking target (such as a person) under the conditions of no influence of light rays, no need of wearing a sensor and no risk of privacy disclosure.
By way of example, the states identifiable by state identification of the tracked targets may include walking, running, sitting, jumping, lying, standing, falling, and the like, covering substantially all common action states.
In some embodiments of the invention, multi-user real-time state monitoring can be realized, and the method has higher accuracy and stronger practicability.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.
It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present invention are not limited to the specific details set forth above, and that these and other objects that can be achieved with the present invention will be more clearly understood from the detailed description that follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention.
Fig. 1 is a schematic flow chart of a millimeter wave-based person status identification method according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a millimeter wave-based personnel state identification process according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a millimeter wave radar signal analysis process in the prior art.
Fig. 4 is a schematic flow chart of a cluster tracking algorithm based on millimeter wave radar.
FIG. 5 is a diagram illustrating a neural network model for state recognition according to an embodiment of the present invention.
Fig. 6 is a block diagram of a finite state machine according to an embodiment of the present invention.
FIG. 7 is a diagram of 3 kinds of edge-out of a finite-state machine according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the following embodiments and accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
It should be noted that, in order to avoid obscuring the present invention with unnecessary details, only the structures and/or processing steps closely related to the scheme according to the present invention are shown in the drawings, and other details not so relevant to the present invention are omitted.
It should be emphasized that the term "comprises/comprising" when used herein, is taken to specify the presence of stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.
The invention provides a millimeter wave-based non-contact state identification method and a corresponding device. Because the state sensing is realized by using the millimeter waves, the action state of the user can be identified in real time without wearing any wearable equipment or sensor by the user in the millimeter wave sensing range, and the identifiable state can cover most common states, such as action states including walking, running, sitting, lying, jumping, standing, falling and the like. Fig. 1 is a schematic flow diagram of a millimeter wave-based person state identification method in an embodiment of the present invention, and fig. 2 is a schematic diagram of a millimeter wave-based person state identification process in an embodiment of the present invention. As shown in fig. 2, in this embodiment, point cloud data is first generated based on an original signal (millimeter wave signal) detected by a millimeter wave radar, where the point cloud data includes original information such as a spatial coordinate of a detection point, doppler data, and a signal-to-noise ratio, and then, clustering is performed based on the point cloud data by using a clustering tracking algorithm to identify each tracked target (e.g., a person to be detected) for target positioning and tracking, then, action classification is performed by using a neural network model (action classification algorithm) for state identification established in the present invention to obtain a tracked target state prediction result (e.g., probability of the target in each state), and then, a final state of the tracked target is determined based on the tracked target state prediction result and a pre-established finite state machine, so as to complete state identification. As shown in fig. 1, the millimeter wave-based person status recognition method according to an embodiment of the present invention includes the following steps S1-S5:
and step S1, acquiring point cloud data in real time based on the millimeter wave signals received by the millimeter wave radar.
The point cloud data may include spatial coordinate data of each detection point, doppler velocity information, signal to noise ratio, and the like.
As an example, the millimeter wave radar is Texas Instruments (TI) millimeter wave radar IWR 6843, which is an FMCW millimeter wave radar, but the present invention is not limited thereto. The millimeter wave radar apparatus has 3 transmission antennas and 4 reception antennas. The millimeter wave radar continuously emits millimeter waves, the millimeter waves emitted by the radar are reflected by the body of a person and then received by a receiving antenna of the radar, and received original signals are subjected to signal processing in the radar to generate point cloud data. The generated click data contains some characteristic values related to the detection point of the tested person.
Fig. 3 is a schematic diagram illustrating an analysis process of a conventional millimeter-wave radar signal. As shown in fig. 3, through the steps of distance processing, Capon beam former, target detection, doppler estimation, and the like of the TI millimeter wave radar IWR 6843, the original signal received by the millimeter wave radar is processed into point cloud data including characteristic values of spatial coordinates x, y, z, doppler velocity information, signal-to-noise ratio, and the like of each detection point. Since fig. 3 is a conventional point cloud data generation process, detailed description thereof is omitted here.
In the embodiment of the present invention, the reflecting radar millimeter wave may be an animal other than a human, a bionic robot, or the like, but the present invention is not limited to these examples, and may be any object to be detected whose state change needs to be detected.
And step S2, tracking the target based on the obtained point cloud data, and determining the space coordinate data of the tracked target.
More specifically, point cloud data are clustered, and the detected person and the point cloud data corresponding to each detected person can be determined based on the clustering result. And positioning and tracking the detected personnel based on the point cloud data of the detected personnel.
Fig. 4 is a schematic flow chart of an algorithm for performing cluster tracking on point cloud data. As shown in fig. 4, the point cloud data of each frame generated by processing is sent to a clustering tracking algorithm, the point of a new frame is associated to a traceable unit or is clustered into a new class through the tracking algorithm, and the position of the target is corrected through an updating algorithm based on extended kalman filtering, so as to realize the distinguishing, positioning and tracking of the tested person. And then outputting target data through a UART serial port, wherein the target data comprises an identification ID (identity) of each detected person and point cloud data, and the point cloud data comprises space coordinates x, y and z of point cloud of a tracking target (detected person), Doppler velocity information of the point cloud, signal-to-noise ratio of the point cloud and the like. After information such as the position, Doppler velocity, signal-to-noise ratio and the like of the tracked target point cloud is obtained, data can be subsequently input into a precession action classification algorithm frame by frame to realize the classification of actions; and further inputting the obtained classification prediction result and the prediction probability into a finite-state machine, and carrying out logic judgment by the finite-state machine according to the self structure to obtain the final state of the target. Therefore, the above data obtained after clustering will be used as a data source of the state recognition algorithm. Since the tracking algorithm is a mature algorithm in the existing point cloud data processing, it is not detailed here.
The present invention can also be used to confirm the number of persons in the sensing area, calculate the flow rate of persons, and the like based on step S2.
Step S3, performing data splitting on the point cloud data related to the tracking target within a time window of a predetermined length by frame, and generating a multi-dimensional data matrix corresponding to each frame of data based on the split data.
The multi-dimensional data matrix includes at least data of a plurality of attributes including a spatial coordinate attribute and at least one of the following attributes: doppler velocity data and signal to noise ratio data.
In the embodiment of the invention, the important solution is how to accurately identify the personnel state by using the target point cloud data extracted by the millimeter wave radar. In order to solve the problem, the point cloud data is split according to space and attributes, a Neural Network (NN) model is designed for extracting the space characteristics and the time sequence characteristics of the point cloud data, and finally accurate state recognition is carried out through a classifier. The point cloud data provided by the invention is split according to the space and the attribute, so that the problem of characteristic ambiguity caused by data hybridization can be solved, and the neural network can extract the characteristics of the point cloud data in the space and time dimensions more efficiently.
The invention further aims to solve the problem of how to maintain the state information of the target under the complex real condition, achieve the purposes of state keeping and transferring, realize the continuous monitoring of the state for a long time and improve the accuracy of state identification. Therefore, the present invention further provides a method for maintaining target State information by using a Finite State Machine (FSM), wherein a State where a user is located is abstracted into nodes of the FSM, actions causing State transition are abstracted into edges of the FSM, the distinguishing problem of the states is classified into a classification problem for the State transition actions, and the State Machine completes State transition by using a logic judgment mode according to the output and self structure of the neural network to obtain a final State of a target, as described in step S5 later. According to the personnel state identification method based on the finite-state machine, the personnel state is abstracted into points of the FSM, the actions causing the state change are abstracted into the edges of the FSM, and the action state information of the personnel is accurately and efficiently maintained.
As an example, in this step, the point cloud data related to the tracking target within the time window with the length of n frames may be subjected to data splitting by frame, and a multi-dimensional data matrix (e.g., a 4-dimensional data matrix) corresponding to each frame data may be generated based on the split data as an input of the pre-trained neural network model, as shown in fig. 5.
The time window is used for maintaining data of each tested person in a preset time period, the size of the time window is n frames, and the time window contains point cloud data information of the latest n frames related to each tested person, for example, the time window comprises information of five attributes including space coordinates x, y and z of the point cloud data, Doppler velocity information and signal to noise ratio. In practice, the value of n is 10, but the invention is not limited thereto, and may be a value greater than or less than 10, such as 5, 15, 20, and the like, and the real-time performance and the detection effect may be balanced based on the actual application scenario. As an example, each quiltThe range of the point cloud cluster of the surveyor is a cube of 2m x 2m (the invention is not limited thereto), the resolution granularity is 4cm (based on the resolution, other values can be also adopted), and each direction has 50 granularity units; taking the space position information of the tracked target of the tested person after the tracked algorithm as the central coordinate of the multidimensional data matrix, and recording the central coordinate as x 0 、y 0 、z 0 . And (3) splitting and sorting the point cloud data information of the detected personnel frame by frame according to the space granularity, and establishing a 50 x 5 4-dimensional data matrix by using the information generated by splitting the point cloud data information frame by frame according to the space granularity and five attributes of the point cloud data. Traversing all point cloud data according to the space coordinates x, y, z and the central coordinate x of each detection point 0 、y 0 、z 0 And (3) calculating the space coordinates (i, j, k) of each detection point in the data matrix, and filling the space coordinates i, j, k, the Doppler velocity and the signal-to-noise ratio into the corresponding position of the data matrix.
In an embodiment of the present invention, the generating a multidimensional data matrix corresponding to each frame of data based on the split data includes: and when the number of the detection points in the point cloud data of the current frame is less than the preset number, filling the blank data by using the part of the detection points of the previous frame closest to the target center point of the current frame until the number of the detection points in the point cloud data of the current frame reaches the preset number. For example, its value is set to 0 at a location where there is no data padding; if the point cloud data of the current frame is too little, a data reuse mode can be adopted to fill the vacant data in the partial point cloud data in the previous frame closest to the target center point of the current frame (with the shortest Euclidean distance) until the number of points reaches a preset number, such as 64.
And step S4, inputting the multidimensional data matrix corresponding to each frame of data into a pre-trained neural network model, and outputting a tracking target state prediction result.
The pre-trained neural network model comprises a multi-layer convolutional layer with the number of channels being the same as the attribute number of data in the multidimensional data matrix, a Long Short-Term Memory (LSTM) network layer and a full-connection classifier; and respectively inputting data of each attribute in the multi-dimensional data matrix corresponding to each frame of data to the multi-layer convolution layer of the corresponding channel, outputting a characteristic sequence corresponding to each frame as the input of the LSTM network layer, fusing the output of the LSTM network layer and inputting the fused output to the full-connection classifier, and outputting a prediction result of the state of a tracked target by the full-connection classifier, wherein the state of the tracked target comprises a plurality of action states. According to the neural network architecture provided by the embodiment of the invention, the convolutional layer is used for extracting the characteristics of each attribute of the point cloud in the space dimension, and the LSTM network is used for extracting the characteristics of the point cloud in the time dimension and accurately classifying the point cloud.
As an example, the multidimensional data matrix (e.g., 4-dimensional data matrix) containing 5 attributes generated in step S3 is input to a 5-channel convolutional layer, the spatial features of the respective attributes of point cloud data in each frame are extracted in a feature extraction manner similar to an RGB image, the data matrix with the 5 attributes of point cloud data is regarded as 5 channels, feature extraction is performed by using the 5-channel convolutional layer pair, the above-mentioned convolution process is performed on each frame of data in a time window, and the 5-channel convolutional layer performs convolution and outputs a total feature sequence X 0 、X 1 …X n-1 (ii) a The convolved characteristic sequence enters an LSTM network layer, and the convolved characteristic sequence X is connected with the action of the tested person in the time dimension because the action of the tested person has continuity 0 、X 1 …X n-1 The input into the LSTM network layer can learn the characteristics of point cloud data in the time dimension and output the time sequence characteristics h 0 ,h 1 ,…,h n-1 (ii) a And performing feature fusion on the output time sequence features of the LSTM network layer, connecting the time sequence features into a vector, inputting the vector into a full-connection classifier for classification, and obtaining a specific classification result and a prediction probability value of the specific classification result, wherein the specific classification result has one of a plurality of action states (the plurality of action states can be selected from part or all of the following states: standing, falling, walking, running, jumping, sitting and lying).
In the embodiment of the present invention, the size n of the time window, the dimension number of the data matrix, the attribute number of the data matrix, the resolution granularity of the point cloud clustering range, and the number of the gap data fillings may be adaptively adjusted as needed, and are not limited to the above examples.
The point cloud data is subjected to space splitting and sorting on the five attributes of x, y, z, Doppler velocity information and signal to noise ratio according to space coordinates, and convolution extraction is respectively carried out on the five attributes, so that not only can the characteristics of the space dimensionality of the point cloud data be extracted, but also the interference among different attributes can be avoided, the correct and efficient extraction of the characteristics of each attribute is ensured, and the characteristic blurring caused by attribute mixture can be avoided.
In an embodiment of the present invention, the method for identifying the state of the tracking target further includes a training step of training the model based on training set data prepared in advance.
And step S5, determining the final state of the tracking target based on the tracking target state prediction result and a pre-established finite state machine.
The personnel state identification method based on the finite state machine abstracts the personnel state into points of an FSM (finite state machine), abstracts actions causing state change into edges of the FSM, can accurately and efficiently maintain the state information of the personnel to be tested, and eliminates inaccurate classification prediction results in the neural network output of state identification.
The pre-established finite state machine contains the transformation relation among the states taking each action state as a node and a state prediction probability minimum threshold. Fig. 6 is a structural diagram of a finite state machine according to an embodiment of the present invention. As shown in fig. 6, the finite state machine defines all possible states of the target in the range of the perception scene, including walking, running, sitting, jumping, lying, standing and falling, and each state corresponds to a node on the finite state machine; the finite state machine also defines the conditions of state transition, as shown in fig. 7, each edge corresponds to a state transition condition, a unidirectional edge from a to B indicates that the state can only be transited from the state a to the state B, a bidirectional edge from C to D indicates that the state C and the state D can be transited to each other under the condition that the conditions are satisfied, a spinning edge from E to E indicates that no state transition action occurs, the state remains unchanged, each edge corresponds to an action and a probability minimum threshold value θ, and the probability minimum threshold value can be set to 0.75 based on experience, but the invention is not limited thereto. The probability minimum threshold is an adjustable parameter and can be adjusted according to needs.
After the output of the neural network model is obtained, the obtained classification prediction result and the prediction probability are input into a finite state machine, and the finite state machine can carry out logic judgment according to the self structure to obtain the final state of the tracking target: and the final state of the tracking target is a state which accords with the transformation relation between the states and has a prediction probability value larger than a preset probability minimum threshold value, or is a state which is not changed. The finite state opportunity checks whether the actions and the probability values corresponding to all outgoing edges of the corresponding nodes of the current state are matched with the output of the obtained neural network model, when the matching is successful (the actions corresponding to the outgoing edges conform to the transformation relation among the states and the prediction probability value is greater than a preset probability minimum threshold), the classification prediction result is considered to be accurate, the state is transferred along the outgoing edges, and the motion state of the detected personnel is output; when all the outgoing edges are mismatched, the state is kept unchanged along the spinning edge, and the state of the output detected person is unchanged.
In the state recognition algorithm, a Finite State Machine (FSM) and a Neural Network (NN) are used as tools, the state of a user is abstracted into nodes of the FSM, actions causing state conversion are abstracted into edges of the FSM, and the problem of distinguishing the states is classified into the problem of classifying the state conversion actions; classifying the actions by using a neural network, taking point cloud data collected by the millimeter wave radar, including position, Doppler, signal to noise ratio and the like, as the input of a neural network module, and giving a classification prediction result and a prediction probability by using the neural network module; and the finite state machine completes state transition by utilizing a logic judgment mode according to the output and the self structure to obtain the final state of the target, so that the real-time state of the user can be identified, including walking, running, sitting, lying, jumping, standing, falling and the like.
By utilizing the millimeter wave-based non-contact state identification method provided by the invention, all tested personnel appearing in a scene carry out state identification according to the following steps: when the tracking unit appears for the first time in the perception scene, the system can locate a node for the tested person in the finite state machine for maintaining the state information of the tested person, and the default of the initial state of the node is set as a station corresponding to the initial state node on the finite state machine. When the acquired data of the tested person reaches the frame number n set by the size of the time window, processing the data related to the tested person and inputting the processed data into the neural network model for action classification, obtaining a prediction result of the action classification and a prediction probability thereof after the data is classified by the neural network model, and taking the output of the neural network model as the input of the finite-state machine; and (3) checking all outgoing edges of the node corresponding to the current state by using a finite state machine, matching the output of the neural network model with the conditions defined on the outgoing edges, if the matching is successful, carrying out state transfer along the edges, and if all the edges are mismatched in matching, spinning, and outputting the final state of the target by using the state machine through the steps.
The invention can avoid the influence of bad environments such as strong and weak light, smog, rain, snow and the like by selecting the millimeter wave with certain penetrating power, and can achieve multi-scene application; the sensor or the wearable device is not needed to be worn, and more comfortable use experience can be provided for the user; a camera does not need to be installed in the sensing area, and privacy disclosure risks do not exist; detectable states include walking, running, sitting, jumping, lying, standing, and falling, covering substantially all common states, which may provide more comprehensive user activity data; because the millimeter wave radar can continuously send FMCW millimeter waves, the state information of a target can be maintained under a complex practical condition, the purposes of state keeping and transferring are achieved, long-time tracking of personnel trajectories and continuous state monitoring are realized, the contingency of discontinuous monitoring is avoided, and more objective user activity data are provided; meanwhile, the method can also realize the real-time state monitoring of multiple persons, has higher accuracy and stronger practicability, and solves some defects existing in the traditional method.
The present invention may be applied in a number of different contexts. For example, these scenarios may include: 1) and analyzing the household activity habits of the family members. For example, the family member's activity track and state can be recorded for a long time, and personalized health guidance suggestions, such as sedentary reminders, are provided for the person through big data analysis to analyze the family member's home activity habits; 2) and the intelligent home is controlled and energy is saved. For example, a light system can be automatically turned on when a user goes home or gets up at night according to the position and the state of the user, an air conditioner or the light system can be automatically turned off when the user sleeps or leaves a room, the gear of the air conditioner and the light intensity information can be intelligently adjusted according to different people numbers and activity states, a more convenient, intelligent and comfortable living environment is provided for the user, meanwhile, the waste of electric power can be prevented, and the effects of energy conservation and environmental protection are achieved; 3) and monitoring the health state of the user. For example, the sleep quality of the user can be analyzed by monitoring the number and time of the user staying at night, so that the health condition of the user can be reflected, and objective data can be provided for diagnosis and treatment of doctors. In addition, the method can also be used for detecting whether family members fall down or not, particularly for families of solitary old people, and the old people can send prompt information to inform families or nursing staff of providing help through software, short messages and the like at the first time after falling down. The above are only some examples of the application scenarios of the present invention, and the present invention is not limited thereto.
In some embodiments of the present invention, the method for identifying a state of a tracking target based on millimeter wave perception further includes: and generating prompt information and/or sending the prompt information to the communication terminal based on the final state of the tracking target determined by the finite state machine. That is, when the final state is judged to be a dangerous state such as a falling state or other states needing prompting, the prompt information can be automatically prompted and/or automatically sent to the communication terminal associated in advance, so that the state can be known at the first time and corresponding measures can be taken, and the effects of guaranteeing the safety of personnel and the like are achieved.
Besides home use, according to the characteristics of no privacy disclosure and strong penetrability, the invention can be combined with other equipment to be used for monitoring the state of personnel in a hospital, such as toilets and wards needing privacy protection, and the like, and can also be used in dressing rooms or other areas needing privacy protection, so that the privacy is protected and rescue is provided for the personnel in time; the invention can also be used in other areas where the camera can not be installed, such as a privacy mechanism, a bank and the like, can realize the state and track detection of the detected personnel entering the sensing area, and can also realize the prompt when the movement track or action of the personnel in the sensing area is detected when the preset condition is not met by combining other equipment.
Correspondingly to the method, the invention also provides a device for identifying the state of the tracked target based on millimeter wave perception, which comprises a computer device and a memory, wherein the memory comprises a processor and a memory, the memory is used for storing computer instructions, the processor is used for executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device realizes the steps of the method.
Embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the foregoing steps of the edge computing server deployment method. The computer readable storage medium may be a tangible storage medium such as Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.
Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and methods described in connection with the embodiments disclosed herein may be implemented as hardware, software, or combinations of both. Whether this is done in hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, plug-in, function card, or the like. When implemented in software, the elements of the invention are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine-readable medium or transmitted by a data signal carried in a carrier wave over a transmission medium or a communication link.
It is to be understood that the invention is not limited to the specific arrangements and instrumentality described above and shown in the drawings. A detailed description of known methods is omitted herein for the sake of brevity. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present invention are not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications and additions or change the order between the steps after comprehending the spirit of the present invention.
Features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments in the present invention.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made to the embodiment of the present invention by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for identifying a target tracking state based on millimeter wave perception is characterized by comprising the following steps:
acquiring point cloud data in real time based on a millimeter wave signal received by a millimeter wave radar, wherein the point cloud data comprises space coordinate data, Doppler velocity information and signal-to-noise ratio of each detection point;
tracking a target based on the obtained point cloud data, and determining space coordinate data of the tracked target;
performing data splitting on point cloud data which is in a time window with a preset length and is related to the tracking target by frames, and generating a multi-dimensional data matrix corresponding to each frame of data based on the split data, wherein the multi-dimensional data matrix at least comprises data of a plurality of attributes, and the data of the plurality of attributes comprises a spatial coordinate attribute and at least one of the following attributes: doppler velocity attributes and signal-to-noise ratio attributes;
inputting a multi-dimensional data matrix corresponding to each frame data into a pre-trained neural network model, and outputting a tracking target state prediction result, wherein the pre-trained neural network model comprises a multi-channel multi-layer convolutional layer with the same channel number as the attribute number of the data in the multi-dimensional data matrix, an LSTM network layer and a full-connection classifier; inputting data of each attribute in a multi-dimensional data matrix corresponding to each frame data into a multi-layer convolution layer of a corresponding channel respectively, outputting a characteristic sequence corresponding to each frame as the input of an LSTM network layer, fusing the output of the LSTM network layer and inputting the fused output into a full-connection classifier, and outputting a prediction result of the state of a tracked target by the full-connection classifier, wherein the state of the tracked target comprises a plurality of action states;
and determining the final state of the tracking target based on the tracking target state prediction result and a pre-established finite state machine.
2. The method of claim 1, wherein the feature sequence corresponding to each frame output by the multi-layer convolutional layer is a one-dimensional feature sequence, and the fully-connected classifier outputs a prediction result of a state of a tracking target as a prediction probability value of each action state.
3. The method of claim 1, wherein the action state is selected from some or all of the following states: standing, falling, walking, running, jumping, sitting and lying.
4. The method of claim 1, further comprising: and training a neural network model.
5. The method of claim 1, wherein the central coordinate of the multidimensional data matrix is a spatial coordinate of a tracking target, and the generating the multidimensional data matrix corresponding to each frame of data based on the split data comprises:
and when the number of the detection points in the point cloud data of the current frame is less than the preset number, filling the blank data by using the part of the detection points of the previous frame closest to the target center point of the current frame until the number of the detection points in the point cloud data of the current frame reaches the preset number.
6. The method of claim 2, wherein the pre-established finite state machine comprises a transformation relationship between states using the action states as nodes and a state prediction probability minimum threshold.
7. The method of claim 6, wherein determining a final state of a tracking target based on the tracking target state prediction result and a pre-established finite state machine comprises:
the final state of the tracking target is a state which accords with the transformation relation between the states and has a prediction probability value larger than a preset probability threshold value, or is a state which is not changed.
8. The method of claim 1, further comprising:
and generating prompt information and/or sending the prompt information to the communication terminal based on the final state of the tracking target determined by the finite state machine.
9. A tracking target state identification device based on millimeter wave sensing, comprising a processor and a memory, wherein the memory has stored therein computer instructions, the processor being configured to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210360722.2A CN114818788A (en) | 2022-04-07 | 2022-04-07 | Tracking target state identification method and device based on millimeter wave perception |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210360722.2A CN114818788A (en) | 2022-04-07 | 2022-04-07 | Tracking target state identification method and device based on millimeter wave perception |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114818788A true CN114818788A (en) | 2022-07-29 |
Family
ID=82534889
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210360722.2A Pending CN114818788A (en) | 2022-04-07 | 2022-04-07 | Tracking target state identification method and device based on millimeter wave perception |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114818788A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115909503A (en) * | 2022-12-23 | 2023-04-04 | 珠海数字动力科技股份有限公司 | Tumble detection method and system based on human body key points |
| CN117158967A (en) * | 2023-07-25 | 2023-12-05 | 北京邮电大学 | Personnel pressure non-sensing continuous monitoring method and system based on millimeter wave sensing |
| CN117503092A (en) * | 2023-10-13 | 2024-02-06 | 中国人民解放军总医院第八医学中心 | ICU delirium risk real-time assessment method based on millimeter waves |
| CN117557977A (en) * | 2023-12-28 | 2024-02-13 | 安徽蔚来智驾科技有限公司 | Environment perception information acquisition method, readable storage medium and intelligent device |
-
2022
- 2022-04-07 CN CN202210360722.2A patent/CN114818788A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115909503A (en) * | 2022-12-23 | 2023-04-04 | 珠海数字动力科技股份有限公司 | Tumble detection method and system based on human body key points |
| CN115909503B (en) * | 2022-12-23 | 2023-09-29 | 珠海数字动力科技股份有限公司 | Fall detection method and system based on key points of human body |
| CN117158967A (en) * | 2023-07-25 | 2023-12-05 | 北京邮电大学 | Personnel pressure non-sensing continuous monitoring method and system based on millimeter wave sensing |
| CN117503092A (en) * | 2023-10-13 | 2024-02-06 | 中国人民解放军总医院第八医学中心 | ICU delirium risk real-time assessment method based on millimeter waves |
| CN117557977A (en) * | 2023-12-28 | 2024-02-13 | 安徽蔚来智驾科技有限公司 | Environment perception information acquisition method, readable storage medium and intelligent device |
| CN117557977B (en) * | 2023-12-28 | 2024-04-30 | 安徽蔚来智驾科技有限公司 | Environment perception information acquisition method, readable storage medium and intelligent device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Luo et al. | Temporal convolutional networks for multiperson activity recognition using a 2-d lidar | |
| Dang et al. | Sensor-based and vision-based human activity recognition: A comprehensive survey | |
| Deep et al. | A survey on anomalous behavior detection for elderly care using dense-sensing networks | |
| CN114818788A (en) | Tracking target state identification method and device based on millimeter wave perception | |
| CN108171278B (en) | Motion pattern recognition method and system based on motion training data | |
| CN108662728B (en) | Information processing method, information processing apparatus, and recording medium | |
| CN110456320A (en) | A kind of ULTRA-WIDEBAND RADAR personal identification method based on free space gait temporal aspect | |
| Jannat et al. | Efficient Wi-Fi-based human activity recognition using adaptive antenna elimination | |
| CN115345908B (en) | Human body posture recognition method based on millimeter wave radar | |
| CN110991559B (en) | Indoor personnel behavior non-contact cooperative sensing method | |
| Li et al. | Collaborative fall detection using smart phone and Kinect | |
| CN115205891A (en) | Personnel behavior recognition model training method, behavior recognition method and device | |
| CN112684430A (en) | Indoor old person walking health detection method and system, storage medium and terminal | |
| Kabir et al. | CSI-IANet: An inception attention network for human-human interaction recognition based on CSI signal | |
| Akilandeswari et al. | Design and development of an indoor navigation system using denoising autoencoder based convolutional neural network for visually impaired people | |
| Jain et al. | Ambient intelligence-based multimodal human action recognition for autonomous systems | |
| Konings et al. | Improved distance metrics for histogram-based device-free localization | |
| Yu et al. | An intelligent implementation of multi-sensing data fusion with neuromorphic computing for human activity recognition | |
| Eyobu et al. | A real-time sleeping position recognition system using IMU sensor motion data | |
| CN116626596A (en) | Social intention recognition method and system based on millimeter wave radar | |
| Chen et al. | Air-CSL: Chinese sign language recognition based on the commercial WiFi devices | |
| Thakur | Vital sign monitoring based on remote PPG and WiFi | |
| CN117520862B (en) | Gesture recognition method based on RFID technology and cutting Fresnel zone principle | |
| Memmesheimer | On the recognition of human activities and the evaluation of its imitation by robotic systems | |
| Sansano Sansano | Machine learning-based techniques for indoor localization and human activity recognition through wearable devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |