CN112597896A - Human body posture imaging method and device based on wireless signals - Google Patents
Human body posture imaging method and device based on wireless signals Download PDFInfo
- Publication number
- CN112597896A CN112597896A CN202011544374.1A CN202011544374A CN112597896A CN 112597896 A CN112597896 A CN 112597896A CN 202011544374 A CN202011544374 A CN 202011544374A CN 112597896 A CN112597896 A CN 112597896A
- Authority
- CN
- China
- Prior art keywords
- human body
- thermodynamic diagram
- network
- image
- posture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/23—Recognition of whole body movements, e.g. for sport training
Abstract
A human body posture imaging method based on wireless signals comprises the following steps: designing an artificial neural network W to regress a wireless signal z to a coordinate value of a key point of a human body: drawing a posture thermodynamic diagram h according to the coordinate values of the human key points: designing a generator network G and a discriminator network D to construct a confrontational learning model of image synthesis; wherein the generator network G comprises an encoder, a series of conditional residual blocks and a decoder; inputting an initial moment image captured by a camera into the encoder, combining a gesture thermodynamic diagram and initial image features by using attention operation PA and embedding the combined image features and the initial image features into an input layer of a conditional residual block, and outputting a fused feature combining the initial image features and the gesture thermodynamic diagram; and inputting the fusion characteristics combined with the initial image characteristics and the posture thermodynamic diagram into the decoder to obtain a target human body posture image.
Description
Technical Field
The invention relates to the crossing field of wireless perception and computer vision, in particular to a human body posture imaging method and device based on wireless signals.
Background
In the field of social security, optical-based cameras are important monitoring devices. However, the shielded object or person often cannot be shot, and considering that the signal (radar or WiFi signal, etc.) emitted by the wireless device has good wall-penetrating performance, it is of great practical significance to monitor the target by synthesizing the optical image with the wireless signal auxiliary camera. Meanwhile, in recent years, there have been many advances in establishing radio sensing systems for sensing and understanding human activities. By analyzing radio signals reflected from the human body, various systems have been devised to track the position, motion and vital signs of the human body. However, the existing method can only roughly estimate the motion information of the human body, and cannot synthesize an optical image at a pixel level.
Disclosure of Invention
In view of the above, the main objective of the present invention is to provide a method and an apparatus for human body posture imaging based on wireless signals, so as to partially solve at least one of the above technical problems.
In order to achieve the above object, as an aspect of the present invention, there is provided a human body posture imaging method based on wireless signals, comprising the steps of:
designing an artificial neural network W to regress a wireless signal z to a coordinate value of a key point of a human body:
drawing a posture thermodynamic diagram h according to the coordinate values of the human key points:
designing a generator network G and a discriminator network D to construct a confrontational learning model of image synthesis; wherein the generator network G comprises an encoder, a series of conditional residual blocks and a decoder;
inputting an initial moment image captured by a camera into the encoder, combining a gesture thermodynamic diagram and initial image features by using attention operation PA and embedding the combined image features and the initial image features into an input layer of a conditional residual block, and outputting a fused feature combining the initial image features and the gesture thermodynamic diagram;
and inputting the fusion characteristics combined with the initial image characteristics and the posture thermodynamic diagram into the decoder to obtain a target human body posture image.
Wherein, the design artificial neural network W represents the coordinate value of the key point of the human body by returning the wireless signal z to the human body as follows:
wherein the loss function of the artificial neural network W is as follows:
wherein the content of the first and second substances,
representing the predicted key point coordinates, p, of the artificial neural network WiRepresenting the true keypoint coordinates and K representing the number of keypoints.
Wherein the postural thermodynamic diagram
Where g (-) represents a Gaussian kernel function.
Wherein the attention operation PA is defined as follows:
PA[fi(xin)，h]＝fi(xin)+α·h·fi(xin)；
wherein x isinIs an image at an initial time, fi(xin) Which represents the image characteristics input to the i-th conditional residual block, alpha is a hyper-parameter for adjusting the magnitude of the attention intensity.
Wherein the loss function of the generator network G is
LG＝LLSG+λ2LL2；
Wherein the content of the first and second substances,
λ2to balance the over-parameters of the loss functions.
Wherein the loss function of the discriminator network D is
LD＝LLSD+λ1LGP
Wherein the content of the first and second substances,
λ1to balance the over-parameters of the loss functions.
As another aspect of the present invention, there is provided a human body posture imaging apparatus based on a wireless signal, for performing the above human body posture imaging method, comprising:
the key point acquisition module is used for designing an artificial neural network W to enable the wireless signal z to return to the coordinate value of the key point of the human body:
the thermodynamic diagram acquisition module is used for drawing a posture thermodynamic diagram h according to the coordinate values of the human key points:
the generator network module is used for designing a generator network G and a discriminator network D to construct an image synthesized confrontation learning model; wherein the generator network G comprises an encoder, a series of conditional residual blocks and a decoder; inputting an initial moment image captured by a camera into the encoder, combining a gesture thermodynamic diagram and initial image features by using attention operation PA and embedding the combined image features and the initial image features into an input layer of a conditional residual block, and outputting a fused feature combining the initial image features and the gesture thermodynamic diagram; and inputting the fusion characteristics combined with the initial image characteristics and the posture thermodynamic diagram into the decoder to obtain a target human body posture image.
Based on the technical scheme, compared with the prior art, the human body posture imaging method and the human body posture imaging device have at least one or part of the following beneficial effects:
the invention combines the related technologies of wireless perception and image synthesis in computer vision, overcomes the limitation that the prior art can only roughly perceive the human body activity under the condition of dark light or object shielding, and can be widely applied to the fields of social security, intelligent home and the like due to the low cost of the device.
Drawings
FIG. 1 is a general block diagram of a model provided by an embodiment of the present invention;
FIG. 2 is a schematic diagram of an attention operation PA provided by an embodiment of the present invention;
FIG. 3 is a W-network predicted human posture thermodynamic diagram provided by an embodiment of the invention;
FIG. 4 is a human body gesture image synthesized by a wireless signal and an initial time image captured by a camera according to an embodiment of the present invention;
FIG. 5 is a visual illustration of a model training process provided by an embodiment of the present invention, wherein the first image is an initial moment image and the last image is a true target body posture image.
Detailed Description
In order that the objects, technical solutions and advantages of the present invention will become more apparent, the present invention will be further described in detail with reference to the accompanying drawings in conjunction with the following specific embodiments.
The invention synthesizes human body posture images at any time by using the wireless signals and the initial time images captured by the camera. The method comprises the following specific steps:
1. designing an artificial neural network W to regress a wireless signal z to a coordinate value of a key point of a human body:
the loss function is as follows:
wherein
Key point coordinates, p, representing the predictions of the network model WiRepresenting the true keypoint coordinates and K representing the number of keypoints.
2. Drawing a posture thermodynamic diagram h according to the predicted key point coordinates:
where g (-) represents a Gaussian kernel function.
3. The generator network G and the discriminator network D are designed to construct a confrontational learning model of image synthesis. Wherein the generator network G consists of an encoder, a series of conditional residual blocks and a decoder.
4. The initial moment image captured by the camera is input to the encoder, and the gesture thermodynamic diagram and the initial image features are combined and embedded into the input layer of the conditional residual block by using the attention operation PA. Attention operation PA is defined as follows:
PA[fi(xin)，h]＝fi(xin)+α·h·fi(xin)
wherein xinIs an image at an initial time, fi(xin) Which represents the image characteristics input to the i-th conditional residual block, alpha is a hyper-parameter for adjusting the magnitude of the attention intensity.
5. And inputting the fusion characteristics combined with the initial image characteristics and the posture thermodynamic diagram into a decoder network to obtain a target human body posture image.
In the above process, the antagonistic learning model is adopted, and therefore the antagonistic learning loss function is designed to train the generator network G and the discriminator network D.
Wherein xr，xfReal and model-synthesized images, respectively.
To stabilize the training, the following loss function was added at the same time:
in summary, the penalty function for the discriminator network D is
LD＝LLSD+λ1LGP
The loss function of the generator network G is
LG＝LLSG+λ2LL2
Wherein λ1，λ2To balance the over-parameters of the loss functions.
The validity of the method of the invention is verified by means of a specific example.
The embodiment verifies the effectiveness of the method provided by the invention on human body posture imaging under the condition that a person exists in the space. In the embodiment, 1 antenna is used for transmitting, and 15 antennas are used for receiving and acquiring wireless signals. FIG. 1 shows a general model structure diagram, which comprises inputting wireless signals and initial images into W network and generator network G encoder, respectively, drawing pose thermodynamic diagram according to W network output key point coordinates, fusing the initial image features output from the encoder in condition residual block by PA operation, inputting into decoder to obtain generated human body pose image, inputting the generated image and real image into discriminator network D by using antagonistic learning model,and performs "true-false" confrontational training on the condition of the posture thermodynamic diagram. For a specific network structure design, the W-network uses 7 convolutional layers and one fully-connected layer. The generator network G comprises an encoder for 3 convolutional layers, a decoder for 3 convolutional layers and 9 conditional residual blocks, each consisting of 2 convolutional layers. For discriminator network D, it consists of 5 convolutional layers. Fig. 2 shows the operation procedure of the attention operation PA. During training, an Adam optimizer is adopted, the learning rate of the front 3/5 training process is set to be 0.002 for the W network, the learning rate of the rear 2/5 training process is set to be 0.001, and the learning rate of the generator network G and the discriminator network D is always 0.0002. In addition, the hyperparameters α, λ1，λ2Set to 0.5, 10.0 and 10.0, respectively. The experimental results are shown in fig. 3 and 4, where fig. 3 is a posture thermodynamic diagram plotted according to the predicted key point coordinates, and fig. 4 is a generated human body posture image. Fig. 5 illustrates the variation of the result of generating images during training, wherein the first image is the initial moment image and the last image is the real target body posture image. Experiments prove that the human body posture imaging result of the method is consistent with the target result.
The invention also discloses a line signal human body posture imaging device, which is used for executing the human body posture imaging method and comprises the following steps:
the key point acquisition module is used for designing an artificial neural network W to enable the wireless signal z to return to the coordinate value of the key point of the human body:
the thermodynamic diagram acquisition module is used for drawing a posture thermodynamic diagram h according to the coordinate values of the human key points:
the generator network module is used for designing a generator network G and a discriminator network D to construct an image synthesized confrontation learning model; wherein the generator network G comprises an encoder, a series of conditional residual blocks and a decoder; inputting an initial moment image captured by a camera into the encoder, combining a gesture thermodynamic diagram and initial image features by using attention operation PA and embedding the combined image features and the initial image features into an input layer of a conditional residual block, and outputting a fused feature combining the initial image features and the gesture thermodynamic diagram; and inputting the fusion characteristics combined with the initial image characteristics and the posture thermodynamic diagram into the decoder to obtain a target human body posture image.
Wherein the loss function of the generator network G is
LG＝LLSG+λ2LL2；
Wherein the content of the first and second substances,
λ2to balance the over-parameters of the loss functions.
Wherein the loss function of the discriminator network D is
LD＝LLSD+λ1LGP
Wherein the content of the first and second substances,
λ1to balance the over-parameters of the loss functions.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A human body posture imaging method based on wireless signals is characterized by comprising the following steps:
designing an artificial neural network W to regress a wireless signal z to a coordinate value of a key point of a human body:
drawing a posture thermodynamic diagram h according to the coordinate values of the human key points:
designing a generator network G and a discriminator network D to construct a confrontational learning model of image synthesis; wherein the generator network G comprises an encoder, a series of conditional residual blocks and a decoder;
inputting an initial moment image captured by a camera into the encoder, combining a gesture thermodynamic diagram and initial image features by using attention operation PA and embedding the combined image features and the initial image features into an input layer of a conditional residual block, and outputting a fused feature combining the initial image features and the gesture thermodynamic diagram;
and inputting the fusion characteristics combined with the initial image characteristics and the posture thermodynamic diagram into the decoder to obtain a target human body posture image.
2. The human body posture imaging method according to claim 1, wherein the designed artificial neural network W represents the coordinate values of the wireless signal z regressing to the human body key points as:
3. the human body pose imaging method according to claim 1, wherein the loss function of the artificial neural network W is as follows:
wherein the content of the first and second substances,
representing the predicted key point coordinates, p, of the artificial neural network WiRepresenting the true keypoint coordinates and K representing the number of keypoints.
4. Human body posture imaging method as claimed in claim 1, characterized in that the posture thermodynamic diagram
Where g (-) represents a Gaussian kernel function.
5. The human body posture imaging method according to claim 1, characterized in that the attention operation PA is defined as follows:
PA[fi(xin)，h]＝fi(xin)+α·h·fi(xin)；
wherein x isinIs an image at an initial time, fi(xin) Which represents the image characteristics input to the i-th conditional residual block, alpha is a hyper-parameter for adjusting the magnitude of the attention intensity.
6. The human body pose imaging method of claim 1, wherein the loss function of the generator network G is
LG＝LLSG+λ2LL2；
Wherein the content of the first and second substances,
λ2to balance the over-parameters of the loss functions.
7. The method of claim 1, wherein the penalty function of the discriminator network D is
LD＝LLSD+λ1LGP
Wherein the content of the first and second substances,
λ1to balance the over-parameters of the loss functions.
8. A human body posture imaging device based on wireless signals, which is used for executing the human body posture imaging method of any one of claims 1-5, and is characterized by comprising the following steps:
the key point acquisition module is used for designing an artificial neural network W to enable the wireless signal z to return to the coordinate value of the key point of the human body:
the thermodynamic diagram acquisition module is used for drawing a posture thermodynamic diagram h according to the coordinate values of the human key points:
the generator network module is used for designing a generator network G and a discriminator network D to construct an image synthesized confrontation learning model; wherein the generator network G comprises an encoder, a series of conditional residual blocks and a decoder; inputting an initial moment image captured by a camera into the encoder, combining a gesture thermodynamic diagram and initial image features by using attention operation PA and embedding the combined image features and the initial image features into an input layer of a conditional residual block, and outputting a fused feature combining the initial image features and the gesture thermodynamic diagram; and inputting the fusion characteristics combined with the initial image characteristics and the posture thermodynamic diagram into the decoder to obtain a target human body posture image.
9. Human body posture imaging device as claimed in claim 8, characterized in that the generator network G has a loss function of
LG＝LLSG+λ2LL2；
Wherein the content of the first and second substances,
λ2to balance the over-parameters of the loss functions.
10. The human body posture imaging device according to claim 8, characterized in that the loss function of the discriminator network D is
LD＝LLSD+λ1LGP
Wherein the content of the first and second substances,
λ1super-parameterisation to balance loss functionsAnd (4) counting.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011544374.1A CN112597896B (en) | 2020-12-23 | 2020-12-23 | Human body posture imaging method and device based on wireless signals |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011544374.1A CN112597896B (en) | 2020-12-23 | 2020-12-23 | Human body posture imaging method and device based on wireless signals |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112597896A true CN112597896A (en) | 2021-04-02 |
| CN112597896B CN112597896B (en) | 2022-09-09 |
Family
ID=75200604
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011544374.1A Active CN112597896B (en) | 2020-12-23 | 2020-12-23 | Human body posture imaging method and device based on wireless signals |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112597896B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115186814A (en) * | 2022-07-25 | 2022-10-14 | 南京慧尔视智能科技有限公司 | Training method and device for confrontation generation network, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10096125B1 (en) * | 2017-04-07 | 2018-10-09 | Adobe Systems Incorporated | Forecasting multiple poses based on a graphical image |
| CN110969124A (en) * | 2019-12-02 | 2020-04-07 | 重庆邮电大学 | Two-dimensional human body posture estimation method and system based on lightweight multi-branch network |
| CN111160085A (en) * | 2019-11-19 | 2020-05-15 | 天津中科智能识别产业技术研究院有限公司 | Human body image key point posture estimation method |
| CN111274921A (en) * | 2019-11-27 | 2020-06-12 | 北京邮电大学 | Method for recognizing human body behaviors by utilizing attitude mask |
| CN111832383A (en) * | 2020-05-08 | 2020-10-27 | 北京嘀嘀无限科技发展有限公司 | Training method of gesture key point recognition model, gesture recognition method and device |
-
2020
- 2020-12-23 CN CN202011544374.1A patent/CN112597896B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10096125B1 (en) * | 2017-04-07 | 2018-10-09 | Adobe Systems Incorporated | Forecasting multiple poses based on a graphical image |
| CN111160085A (en) * | 2019-11-19 | 2020-05-15 | 天津中科智能识别产业技术研究院有限公司 | Human body image key point posture estimation method |
| CN111274921A (en) * | 2019-11-27 | 2020-06-12 | 北京邮电大学 | Method for recognizing human body behaviors by utilizing attitude mask |
| CN110969124A (en) * | 2019-12-02 | 2020-04-07 | 重庆邮电大学 | Two-dimensional human body posture estimation method and system based on lightweight multi-branch network |
| CN111832383A (en) * | 2020-05-08 | 2020-10-27 | 北京嘀嘀无限科技发展有限公司 | Training method of gesture key point recognition model, gesture recognition method and device |
Non-Patent Citations (3)
| Title |
|---|
| GLORIA ZEN等: "Learning Personalized Models for Facial Expression Analysis and Gesture Recognition", 《 IEEE TRANSACTIONS ON MULTIMEDIA 》 * |
| 王冉: "基于深度卷积神经网络的人体姿势估计研究", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
| 许忠雄: "监控视频实时多人姿态估计算法研究与应用", 《中国优秀硕士学位论文全文数据库（电子期刊）》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115186814A (en) * | 2022-07-25 | 2022-10-14 | 南京慧尔视智能科技有限公司 | Training method and device for confrontation generation network, electronic equipment and storage medium |
| CN115186814B (en) * | 2022-07-25 | 2024-02-13 | 南京慧尔视智能科技有限公司 | Training method, training device, electronic equipment and storage medium of countermeasure generation network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112597896B (en) | 2022-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | CEAM-YOLOv7: Improved YOLOv7 based on channel expansion and attention mechanism for driver distraction behavior detection | |
| CN114862837A (en) | Human body security check image detection method and system based on improved YOLOv5s | |
| Jacob et al. | Depth information enhancement using block matching and image pyramiding stereo vision enabled RGB-D sensor | |
| CN112597896B (en) | Human body posture imaging method and device based on wireless signals | |
| An et al. | Fast and scalable human pose estimation using mmwave point cloud | |
| Zheng et al. | Recovering human pose and shape from through-the-wall radar images | |
| CN115526983A (en) | Three-dimensional reconstruction method and related equipment | |
| JP2023539865A (en) | Real-time cross-spectral object association and depth estimation | |
| CN113724308B (en) | Cross-waveband stereo matching algorithm based on mutual attention of luminosity and contrast | |
| CN116057587A (en) | Living body detection method, training method of living body detection model, device and system thereof | |
| Zhang et al. | Hawk‐eye‐inspired perception algorithm of stereo vision for obtaining orchard 3D point cloud navigation map | |
| Xing et al. | Multi-level adaptive perception guidance based infrared and visible image fusion | |
| CN114782911B (en) | Image processing method, device, equipment, medium, chip and vehicle | |
| Xie et al. | Accurate human pose estimation using rf signals | |
| CN116453160A (en) | Suspicious article non-stop detection method and system based on millimeter wave imaging | |
| CN110348355A (en) | Model recognizing method based on intensified learning | |
| US20230103112A1 (en) | System and method for monitoring activity performed by subject | |
| Cho et al. | Multistage fusion with dissimilarity regularization for SAR/IR target recognition | |
| Lin et al. | Enhancing deep-learning object detection performance based on fusion of infrared and visible images in advanced driver assistance systems | |
| Xiao et al. | Introduction to image fusion | |
| CN110781764B (en) | Intelligent microwave sign language identification method | |
| Xie et al. | RPM 2.0: RF-based Pose Machines for Multi-Person 3D Pose Estimation | |
| CN117037221B (en) | Living body detection method, living body detection device, computer equipment and storage medium | |
| Jin et al. | A Hybrid Model for Object Detection Based on Feature-Level Camera-Radar Fusion in Autonomous Driving | |
| Jonasson et al. | Structured light based depth and pose estimation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |