WO2020177498A1 - Non-intrusive human body thermal comfort detection method and system based on posture estimation - Google Patents

Abstract

From the perspective of posture estimation, a new non-intrusive human body thermal comfort detection method is disclosed and provided. First, 12 human body thermal comfort postures are obtained through definition and verification by means of a questionnaire survey method; and then, results of posture estimation and human body thermal comfort detection are output by means of computer video collection, image preprocessing, depth image processing, training testing and delivery and application, wherein the depth image processing comprises locking a face region and a point region based on bone nodes and setting an action recognition determination condition and threshold corresponding to different posture estimations, and the posture estimations are obtained by comparing changes in the coordinates of the bone nodes in previous and next frames. When the detection method in the present invention is applied to an intelligent building or a vehicle, an effective feedback signal can be provided for a heating, ventilation and air-conditioning system in real time, such that people in a scenario feel more comfortable, and energy is reliably economized.

Description

Non-invasive human thermal comfort detection method and system based on attitude estimation Technical field

The invention relates to technologies such as human thermal comfort non-invasive detection, posture estimation, energy efficiency, artificial intelligence, etc., in particular to a non-invasive human thermal comfort detection method based on posture estimation, and belongs to the field of computer science and HVAC technology.

Background technique

Real-time perception of human thermal comfort is essential for building energy conservation or automobile energy conservation. If the current thermal comfort state of indoor/car occupants can be grasped in real time, then effective feedback signals can be provided to the central air conditioning system in real time to control the temperature, humidity and airflow parameters of the entire room/car; not only can it meet the thermal comfort of the human body Demands can also achieve the goal of building energy conservation, thereby serving the "people-oriented" smart building requirements.

According to the U.S. Energy Information Administration (EIA: USEnergy Information Administration) report, building energy consumption accounts for 21% of the world’s energy consumption, and in all building energy consumption, central air conditioning systems (HVAC: heating ventilation air condition) ) The consumption accounted for 50%. It can be seen that the detection of human thermal comfort and the adjustment of the response output of HVAC by this are of great significance to energy conservation.

At present, there are three methods for detecting human thermal comfort, namely questionnaire survey method, environmental monitoring method and physiological detection method. Among them, physiological detection methods are subdivided into three types: invasive, semi-invasive and non-invasive. The questionnaire survey method requires continuous feedback from users and has low operability; the environmental monitoring method mainly detects objective parameters such as indoor temperature, humidity and airflow, and assumes that within a certain threshold range, most people will be satisfied. The physiological detection method can detect the thermal comfort of a person in real time, but the practicability of the invasive and semi-invasive methods is low because they both need to attach the sensor to the body. The non-invasive method is highly feasible and is currently the focus of research.

Based on the current technological level, the construction industry has adopted an environmental monitoring method. For related equipment, such as heating devices and cooling devices, there are knobs that allow users to adjust themselves according to their thermal comfort. Therefore, there are two major problems: 1) According to the International Organization for Standardization (ISO: International Organization for Standardization) and the American Society of Heating, Refrigeration and Air-Conditioning Engineers (ASHRAE: American Society of Heating, Refrigeration and Air-Conditioning Engineers) In the definition of "environment", 80% of people are satisfied, and it can be judged as a thermally comfortable environment, so the environmental monitoring method is difficult to meet the needs of more people or everyone; 2) The user's self-adjusting knob lacks automation. Neither of these two points are in line with the "people-oriented" thinking.

The rapid development of computer vision and machine learning provides the conditions for seeking technical solutions for non-invasive detection of human thermal comfort. The original intention and essence of computer vision is to allow machines to understand the world. Since the emergence of deep learning technology, it can be competent for many classification and prediction tasks under the training of massive data, which provides the possibility for non-invasive human thermal comfort detection.

Summary of the invention

In view of the above-mentioned current state of insufficient energy consumption control in the prior art, the purpose of the present invention is to propose a novel non-invasive human thermal comfort detection method and system from the perspective of human body posture estimation in a scene. This provides accurate and effective feedback signals for the central air conditioning system (HVAC) in real time, making the scene feel more comfortable and saving energy.

In order to achieve the above-mentioned first objective, the technical solution of the present invention is: a non-invasive human thermal comfort detection method based on posture estimation, including the steps:

S1. Define several postures related to the thermal comfort of the human body, and verify the validity through questionnaire survey;

S2: Video acquisition and preprocessing. The computer vision device captures and collects image data for subjects, and preprocesses and outputs pictures of interest domains;

S3. Generate a basic parameter matrix, capture the coordinates of the whole body skeleton node for each frame of interest domain picture input, and filter frame by frame according to the confidence value;

S4. Retrieval area, according to the action recognition requirements of the defined pose estimation, search and delimit several areas related to the detection part one by one in the interest domain pictures obtained by screening, and the delineation of the area is based on the coordinates of the skeleton node in the basic parameter matrix ；

S5. Lock bone key points, and lock one or more bone node coordinates related to pose estimation in each area corresponding to each frame of interest domain picture;

S6. Condition judgment, set the action recognition judgment conditions and thresholds corresponding to different posture estimations, and compare the change trends of the coordinates of each bone node in the previous and subsequent frames;

S7. Output the posture and detection result, recognize the posture according to the corresponding relationship between the coordinate change of the bone node and the judgment condition and the threshold, and output the thermal tendency and thermal comfort level correspondingly.

Preferably, the posture related to the thermal comfort of the human body in step S1 of the detection method includes: wiping sweat and hand fan corresponding to the first thermal comfort level, shaking the chest T-shirt and scratching the head corresponding to the second thermal comfort level, and corresponding to the third thermal comfort level. Thermal comfort level roll-up sleeves, corresponding to the fourth thermal comfort level walking, corresponding to the fifth thermal comfort level shrinking shoulders, corresponding to the sixth thermal comfort level arms, legs crossed, hands on the neck, and the seventh thermal comfort level His hands breathed and stomped.

Preferably, the image data in step S2 of the above detection method is 30 frames/second continuous frames, and the preprocessing process is to extract pictures containing human body postures frame by frame, remove picture noise and enhance, and output the interest domain picture according to the area of the human body .

Preferably, in step S3 of the above detection method, by calling the OpenPose platform or the bone capture algorithm, the coordinates and confidence values of the whole body bone nodes are directly obtained for each frame of interest domain image input, and an i(x, y, ε) parameter is output , The whole body bone nodes include nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye , Left eyes, right ears, and left ears are numbered in the order of 0-17, and the picture background is numbered 18. i represents the bone node and its corresponding number, x and y respectively represent the position of each bone node in the image coordinate system of the interest domain The coordinate value, ε represents the confidence value and 0.5 is set as the screening threshold, n frames of interest domain pictures with ε≥0.5 are adopted and the interest domain pictures with ε<0.5 are discarded.

Preferably, the detection method described above is based on the action recognition requirements of the defined posture estimation, the area in step S4 area locking includes the forehead, the top of the head, the left side of the head, the right side of the head, the intersection area between the left side of the face and above the chest. The crossing area on the right side of the face and above the chest, the area between the neck and the wrists, the chest and abdomen areas, the shoulders, elbows and the areas between the wrists, the hips, knees and the areas between the ankles.

Preferably, the detection method described above is based on several postures defined in step S1, the action corresponding to each posture is associated and matched with a part of the whole body bone nodes, and the point field in step S5 locks a part of the bone node coordinates associated with a specific posture.

Preferably, the judgment condition in step S6 of the above detection method includes one or several combinations of the concentrated area where the displacement changes of more than one bone key points involved in the posture estimation, the relative distance, the frequency, and the range of the distance change in the previous frame interval; The threshold value is an empirical value after the training test, and includes an associated relative distance threshold value and a frequency threshold value respectively.

Preferably, the above detection method further includes a posture detection and verification step S8. The subject makes random actions to the computer vision device according to the posture defined in step S1, and verifies the conformity of the output thermal tendency and thermal comfort level with the posture .

In order to achieve the above-mentioned second objective, the technical solution of the present invention is: a non-invasive human thermal comfort detection system based on posture estimation, which is realized by a computer and several pre-defined postures related to human thermal comfort, including:

Video acquisition and preprocessing unit, computer vision device to capture image data facing the subject, and used to preprocess and output interest domain pictures in the processor;

Generate the basic parameter matrix unit, which is used by the processor to capture the coordinates of the whole body skeleton node for each frame of interest domain picture input, and screen it frame by frame according to the confidence value;

The search area unit is used for the processor to search and delimit several regions related to the detection part one by one according to the action recognition requirements of the defined posture estimation. The delineation of the regions is based on the skeleton in the basic parameter matrix. The coordinates of the node;

The bone key-point locking unit is used for the processor to lock one or more bone node coordinates related to pose estimation in each area corresponding to each frame of interest domain picture;

The condition judgment unit is used for the processor to set the action recognition judgment conditions and thresholds corresponding to different posture estimations, and compare the change trend of the coordinates of each skeletal node in the previous and subsequent frames;

The output posture and detection result unit is used for the processor to recognize the posture according to the corresponding relationship between the coordinate change of the skeleton node and the judgment condition and threshold, and correspondingly output the thermal tendency and thermal comfort level of the detected human body.

Compared with the prior art, the present invention has outstanding substantive features and significant advancement, which is shown as follows:

1. Convenient application. The detection method does not need to wear or carry the sensor close to the body, and it is convenient and practical to use video capture and posture estimation.

2. People-oriented, can meet the thermal comfort experience of all employees in the scene, and achieve the effect of adjusting the environment to serve people.

3. Save energy, provide effective feedback signals for HVAC in real time, and have predictive functions through data accumulation, so as to achieve all-round energy saving in rooms, buildings, and communities.

Description of the drawings

Fig. 1 is a schematic diagram of an example of the thermal response posture defined by the detection method of the present invention.

Figure 2 is a schematic diagram of an example of a cold reaction posture defined by the detection method of the present invention.

Fig. 3 is a schematic diagram of the distribution of bone key points in the detection method of the present invention.

Fig. 4 is a graph of the results of a questionnaire survey of defined thermal comfort attitude in step S1 of the detection method of the present invention.

Fig. 5 is an algorithm flow chart of the non-invasive human thermal comfort detection method of the present invention.

detailed description

The many shortcomings of the existing technology for human thermal comfort detection methods and the obvious application experience of HAVC system control fixation or manual participation are investigated. The invention relies on the development of computer vision and machine learning, and is committed to real-time perception of human comfort for heating and cooling systems, thereby providing real-time and effective feedback signals to participate in the automatic operation of the temperature regulator; it may meet the thermal comfort requirements of users to the greatest extent, and finally achieve People-oriented and energy saving in the true sense.

To this end, the present invention opens up a brand-new branch of scientific research, and innovatively proposes a non-invasive human thermal comfort detection method and system based on attitude estimation. Its technical realization is: according to Fanger’s theory, 12 thermal comfort evaluations are defined A large number of questionnaire surveys have proved the rationality of the defined posture; and on this basis, a human thermal comfort detection algorithm based on posture estimation is designed, which contains multiple sub-algorithms to recognize these actions and use the received The tester conducts a test to verify the rationality of the test algorithm, and outputs the corresponding results for reference by the HAVC system.

In a step-by-step overview, the non-invasive human thermal comfort detection method includes: S1, defining a number of postures related to human thermal comfort, and verifying the validity through a questionnaire survey; S2, video capture and preprocessing, computer vision device Shoot and collect image data (30 frames/second continuous frames) for the subject, and preprocess and output the interest domain picture; S3, generate a basic parameter matrix, capture the coordinates of the whole body bone node for each input frame of interest domain picture, and Screening frame by frame according to the confidence value; S4. Retrieval area, according to the defined gesture estimation action recognition requirements, search and delimit several areas related to the detection part one by one in the interest domain images obtained by screening, and the basis of the delimitation is basic The coordinates of the bone nodes in the parameter matrix; S5, lock the key points of the bones, and lock one or several bone node coordinates related to the pose estimation in each area corresponding to each frame of the interest domain picture; S6, the condition judgment, set the corresponding different poses Estimated action recognition judgment conditions and thresholds, compare the change trend of the coordinates of each bone node in the previous and subsequent frames; S7, output the posture and detection results, recognize the posture according to the correspondence between the change of the bone node coordinates and the judgment conditions and thresholds, and output heat accordingly Tendency and thermal comfort rating.

The following is a more specific step-by-step detailed understanding: As shown in Figure 5, first step S1, people will have different postures under cold and hot conditions. Therefore, it is an effective way to estimate thermal comfort based on posture. Looking for a gesture with commonality has far-reaching significance. Shown in Figure 1 and Figure 2 shows the heat and cold reaction attitude change law. Based on the Fanger theory, the present invention defines 12 thermal postures, and a large number of questionnaire surveys prove their effectiveness. The types of postures include wiping sweat and hand fan corresponding to the first thermal comfort level, shaking chest T-shirt and head scratching corresponding to the second thermal comfort level, rolled sleeves corresponding to the third thermal comfort level, and corresponding to the fourth thermal comfort level Walking corresponds to shoulder shrinkage corresponding to the fifth thermal comfort level, arms folded, crossed legs, hands on the neck corresponding to the sixth thermal comfort level, and hands breathing and stomping corresponding to the seventh thermal comfort level. The thermal comfort state represented by each posture is shown in the following table.

No.	Human posture	Scoring	Thermal comfort level	Detection method recognition ability
1	Wipe sweat	3	heat	can
2	Hand fan	3	heat	can
3	T-shirt shaking chest	2	warm	can
4	Scratch your head	2	warm	can
5	Roll up sleeves	1	Slightly warm		can
6	walk	0	neutral	can
7	Shrink shoulders	-1	Slightly cold	can
8	Arms folded	-2	cool	can
9	Legs crossed	-2	cool	can
10	Put your hands on your neck to keep warm	-2	cool	can
11	Breathe	-3	cold	can
12	Stomp	-3	cold	can

The specific operation of the 12 posture verification is to invite 400 subjects to conduct a questionnaire survey on the proposed 12 postures, and a total of 369 valid questionnaires were harvested. In his view, the posture is a thermal response posture and a cold response posture. Or "Neither". And statistically, the result chart shown in Figure 4 is obtained. The result shows that the defined posture conforms to people's understanding of heat and cold. Of course, subjects of different regions and ages will have deviations in their understanding of posture, but the overall difference can be ignored, and the scale of subjects can be expanded to thousands or ten thousand, just by increasing the workload to optimize the verification results.

On the basis of posture definition, the human thermal comfort detection method needs to design a complete algorithm architecture to realize posture estimation and detection result output. It mainly includes video acquisition and preprocessing, basic parameter matrix generation, region retrieval, bone key point locking, condition judgment, gesture output, etc. Through this algorithm architecture, posture estimation, thermal tendency recognition and thermal comfort level recognition can be realized. Corresponding to different postures, due to the difference of judgment conditions, the architecture is divided into multiple sub-algorithms.

The specific embodiments of the present invention will be described below with reference to the drawings and technical solutions.

S2. Video acquisition and preprocessing. The data input of the present invention is mainly video images. The common vision sensor associated with computer communication collects data at a rate of 30 frames per second, and the collected data realizes three preprocessing. 1) Frame-by-frame extraction: Mainly realize the extraction of pictures of each frame, and 1800 pictures containing poses are imported into the detection system every minute; 2) Denoising and enhancement: On the basis of the original picture, noise such as salt and pepper noise is removed, and The picture is enhanced to facilitate subsequent search for bone nodes; 3) Search for interest areas: In order to reduce the computational load of subsequent algorithms, the present invention only searches for areas of information value, and defines such valuable areas as interest areas ( ROI: Region of Interest), output the interest region picture backward; and the main basis for determining the interest region is to determine the region where the human body is located.

S3. Generate a basic parameter matrix. The main task of this step is to output the coordinates of the bone node, and according to the confidence value, discard unreliable frames and retain the credible frames for subsequent posture analysis. Here, this solution chooses to call the OpenPose platform or other bone capture algorithms to directly obtain bone nodes and confidence values. The algorithm proposed by the present invention (referred to as the NIMAP method in English) is a general framework, and bone extraction is one of the sub-modules. Of course, the sub-module for extracting bones can also obtain the coordinates of bone nodes through massive data training by constructing a deep learning network. However, this solution is relatively independent and serves as a secondary alternative measure in order to avoid primary and secondary conflicts with the algorithm of the present invention.

As shown in Figure 3, it is the key points of human bones output in this step. Numbering in the order of 0-18, a total of 17 bone nodes are labeled, and the number 18 represents the background of the collected picture. These bone nodes are nose 0, neck 1, right shoulder 2, right elbow 3, right hand 4, left shoulder 5, left elbow 6, left wrist 7, right hip 8, right knee 9, right ankle 10, and left hip. 11. Left knee 12, left ankle 13, right eye 14, left eye 15, right ear 16, left ear 17.

Each frame of interest domain picture, after input into this module, will output a parameter of i (x, y, ε), where i represents the bone node and its corresponding number, and x and y respectively represent each bone node in the interest domain picture The coordinate value in the coordinate system, ε represents the confidence value. According to the acquisition rate (30 frames/second) of the visual sensor of the present invention, 1800×18×3 bone node data are generated in one minute.

In the recognition of bone key points, some pictures will be displaced due to algorithm errors. For this reason, the present invention sets the confidence value ε, and the related threshold value is set as follows: ε>=0.5 is adopted; ε<0.5 is discarded.

Here 0.5 is the empirical value set during the test of the present invention. That is, when ε is greater than or equal to 0.5, the corresponding ROI picture is adopted, and the output parameters are also adopted. Assuming that in one minute of data, n frames are adopted, the corresponding "basic parameter matrix" is n×18×3.

S4. Retrieve the area, that is, lock the area generated by the related action. For different postures, the present invention sets different areas. For different postures, the judgment area is explained.

Wiping sweat: No recognition of sweat wiping actions outside of the head. Therefore, the sweat-wiping area is set as the forehead.

Hand fan wind: Considering the two possibilities of left hand and right hand, two sets of areas are set. They are the intersection area on the left side of the face and above the chest; the intersection area on the right side of the face and above the chest.

Shaking the chest T-shirt: The area is set as the rectangular area between the crotch and the four points of the shoulder, that is, the area under the jurisdiction of the chest and abdomen.

Head scratching: Set three areas on the top of the head, the left side of the head and the right side of the head.

Roll up sleeves: set in the area between the shoulder and the wrist.

Walking and stomping: For walking and stomping, due to the significant regional overlap, a sub-algorithm is used. The setting area includes knees, neck, shoulders and hips.

Shrink shoulders: set the shoulders, ankles and hip areas.

Arm arms: Set the elbow and wrist area.

Legs crossed: the ankle and knee area.

Put your hands on your neck to keep warm and your hands breathe: Also in view of the significant area overlap, set the wrist and neck area.

Locking the above-mentioned areas is mainly based on the "basic parameter matrix" output in the previous step, especially the coordinate values of the bone nodes, and the relative position of the set area can be determined through (x, y).

S5. Lock the key points of the bones. The present invention gradually performs gesture recognition according to steps from large to small. The function of step S4 is to lock the area of gesture recognition. On this basis, this step focuses on locking the key bone points corresponding to a certain gesture, and the locking method is based on the coordinate parameters obtained in step S3. It is understandable that step S4 is regarded as area locking, and step S5 is dot area locking, which involves a specific one or several points in the above domain. The key points of the skeleton defined by the relevant posture are as follows.

Wiping sweat: involving the left and right hands respectively wiping sweat, a total of two groups of key bone points, one is the left wrist (number 7), the right eye (number 14); the other is the right wrist (number 4), the left eye (number 15).

Hand fan wind: involves left and right hand fan wind, a total of two groups of bone key points, one is left wrist (number 7), left elbow (number 6); the other is right wrist (number 4), right elbow (No. 3).

Shaking the chest T-shirt: The key bone points are the wrists (numbers 4 and 7), the elbows (numbers 3 and 6), and the ears (numbers 16 and 17).

Head scratching: The key points of the bones are the wrists (numbers 4 and 7) and ears (numbers 16 and 17).

Roll the sleeves: the key points of the bones are the wrists (numbers 4 and 7), the elbows (numbers 3 and 6), and the shoulders (numbers 2 and 5).

Walking and stomping: The key bone points are the knees (numbers 9 and 12) and ankles (numbers 10 and 13).

Shrinking the shoulders: The key points of the skeleton are the wrists (numbers 4 and 7), hips (numbers 8 and 11), ankles (numbers 10 and 13), and shoulders (numbers 2 and 5).

Arm arms: The key points of the bones are the elbow (numbers 3 and 6) and the wrist (numbers 4 and 7).

Legs crossed: The key points of the skeleton are the wrists (numbers 4 and 7) and knees (9 and 12).

Put your hands on your neck to warm your hands and breathe: The key points of the bones are the neck (number 1), wrists (number 4 and 7) and nose (number 0).

S6. Conditional judgment: After the aforementioned two steps of locking, observe one or several key bone points, analyze its motion law, compare the difference between the previous and the next frame, and distinguish different postures through relevant threshold limits. In this step, different judgment conditions are set mainly for different postures.

The human body will naturally adjust its posture so that it is in a thermally comfortable state. The movement of these bones and joints will produce various displacement changes in the space around the body. Let sp _i denote the key points of human bones. Since the image captured by the visual sensor is two-dimensional information, sp _i can be expressed as sp _i =[x _i , y _i ], i = 1,..., k ( 1); where x _i and y _i represent the abscissa and ordinate in the image coordinate system, the variable i represents different bone key points and their corresponding numbers, and the variable k represents the maximum value of bone key points. In the present invention, k = 18. If sp _i can be captured accurately, then a sub-algorithm can be constructed to recognize various human postures. Through this step, the digitization of human bones is completed.

In order to simplify the complexity of technical processing, the present invention designs it in a sub-algorithm based on the commonality of walking and stomping, and distinguishes between the sub-algorithms according to their differences. In the same way, hands on the neck to keep warm and hands breathing are also combined in a sub-algorithm. The 12 posture related actions correspond to 10 sub-algorithms.

In order to facilitate the recognition of various gestures, the present invention defines several variables. Explained as follows:

First, in order to calculate the Euclidean distance L between bone joint points, the present invention defines a standard distance L _s , namely: L _s =|sp ₇ -sp ₆ | (2), where sp ₇ represents the left wrist, sp ₆ represents the left elbow. Based on formula (2), the relative distance is calculated as:

According to different postures, the present invention will calculate different relative distances L _r and set different condition thresholds. In addition, parameters L _{r_max} and L _{r_min} are introduced to represent the maximum and minimum values of L _r respectively. It should be noted that L _{r_max} and L _{r_min} need not be set for every posture, depending on the specific situation. In the design of the sub-algorithm, the slope and the change of the x and y coordinate values between different frames are also designed. The flowchart of the entire algorithm is shown in Figure 5, and the judgment conditions for each action are described as follows.

For the wiping action, two distances are solved, which are the distance between the left wrist and the right eyebrow Ls ₁ =|sp ₇ -sp ₁₄ |, the distance between the right wrist and the left eyebrow Ls ₂ =|sp ₄ -sp ₁₅ |. Solve the corresponding relative distances L _r1 and L _r2 according to formula (3). After the relative distance is obtained, if the relative distance is less than 1.8, it is judged as a wiping action, that is, L _r1 <1.8 or L _r2 <1.8. It should be noted that 1.8 here is a relative value without a unit and is a test training The experience value obtained during the process.

For the hand fan action,

Judgment condition 1: Generally, reciprocating movement is performed in an area. Set the upper and lower limits of its reciprocating motion to 120 and 8, that is, L _{r_max} =120 and L _{r_min} =8.

Judgment condition 2: In addition to the judgment of the upper and lower thresholds, for the hand fan, the calculated relative distance must be constantly changing. Therefore, the mechanism used is: continuous observation for 2 seconds, because the sampling rate is 30 frames/second, so it is actually continuous observation of 60 frames of pictures, if the number of changes (frequency m) is greater than (60÷2.5), that is, m> (60÷2.5), it may be a hand fan. It should be noted that 2.5 is also the experience value obtained during the test training process.

Judgment condition 3: It is also necessary to judge whether the hand is under the ear.

Therefore, if all the above three conditions are met, that is, m>(60÷2.5), L _r ∈ [8,120], and the hand is under the ear, it can be judged as a hand fan.

For the shaking of the chest T-shirt action,

Judgment condition 1: consistent with the hand fan, a relative distance upper and lower limit is set, which is the range of the hand shake T-shirt, even if L _r ∈ [8,120].

Judgment condition 2: Continuous observation of data for 2 seconds, and m>(60÷2), it is considered that it may be a jittery T-shirt. In other words, the relative distance has continuously changed more than 30 times within 2 seconds. The 2 here is also the experience value obtained after the training test.

Judgment condition 3: Detect the distance L _rj between the wrist and the ear. If the relative distance is less than 1.8, it is considered not to be the action of "shaking the chest T-shirt". In other words, L _rj ≥ 1.8, it may be a t-shirt shaking the chest.

Therefore, when all the above three conditions are met, that is, m>(60÷2), L _r ∈ [8,120], and L _rj ≥ 1.8, it can be determined that the action is a t-shirt shaking the chest.

There are three situations for head scratching, namely the left side of the head, the right side of the head and the top of the head.

Scratching the left side of the head: Judging condition 1: The distance between the left wrist and the left ear is less than 1.8, and the left wrist is above the nose, and the left wrist is on the left side of the left ear; Judging condition 2: The distance between the wrists and eyes on both sides is greater than 1.8. If the above two conditions are met at the same time, it is judged as a head-scratching action.

Scratching the right side of the head: Judging condition 1: The distance between the right wrist and the right ear is less than 1.8, and the right wrist is above the nose, and the right wrist is on the right side of the right ear; Judging condition 2: The distance between the wrists and eyes on both sides is greater than 1.8. If the above two conditions are met at the same time, it is judged as a head-scratching action.

Head scratching: Judgment condition 1: The distance between the right wrist and the right ear is greater than 1.8, the right wrist is above the nose, and the right wrist is between the ears; Judgment condition 2: The distance between the left wrist and the left ear is greater than 1.8, and the left wrist is on the nose Above, with the left wrist between the ears. If condition 1 or condition 2 is met, it can be judged as scratching the head.

For the action of rolling the sleeves, first judge whether the wrist is in the corresponding position of the sleeves. The method is as follows: 1) The left wrist is between the "right wrist and the right shoulder", which is mainly realized by the comparison of the y coordinate, and between the left wrist and the right elbow. The distance is less than or equal to 0.9. Or 2) The right wrist is between "left wrist + left shoulder", which is also realized by comparing the y coordinate, and the distance between the right wrist and the left elbow is less than or equal to 0.9.

Secondly, when the wrist is in the area corresponding to the sleeve, the present invention will judge the change between the previous and subsequent frames, and the focus is to judge the difference in the y coordinate of the wrist. The basic mechanism is: compare the y-coordinates of the wrist in five frames before and after to determine whether the wrist is moving along the arm. Then, between two frames, the change range of the y coordinate must be greater than 10, if this condition is met, it is judged as the sleeve-rolling posture. Similarly, 0.9 and 10 are the experience values obtained after the training test.

For walking and stomping, the steps include:

Determine whether the legs are straight or bent-

Define line 1: Define the two points of the hip and knee, connect them to line 1, and calculate its slope K1;

Define Straight Line 2: Define the two points of knee and ankle, connect them to Straight Line 2, and calculate its slope as K2;

If the slopes of line 1 and line 2 are close (K1≈K2), the leg is considered to be in a straightened state;

If the slope of the straight line 1 and the straight line 2 differ greatly (K1-K2>30), the leg is considered to be in a bent state, where 30 is also the empirical value obtained after the training test.

Continuous frame judgment, intercept 2 seconds of image for continuous judgment. If the trend of straight and bend changes 30 times, it is considered that the first condition of stomping judgment is satisfied; if it is less than 30 times, it is not stomping. The 30 and 2 seconds here are the empirical values obtained after the algorithm test. Simultaneously observe whether there are continuous changes in five key bone points (neck, left shoulder, right shoulder, left hip, right hip) for 2 seconds.

The final judgment of walking and stomping. If the positions of the 5 points are different, then it can be judged as walking. The threshold of the position difference between these 5 points is set to 10 (empirical value); if the slope of the legs changes continuously and the above 5 points If the positions are the same, the second condition of the stomping judgment is satisfied. Together with the first judgment condition, if both are satisfied, it can be considered as a stomping.

For shoulder reduction,

Judging condition 1: Judging that the hands are close to the hips and the legs are close together——

Specifically, the distance between the left wrist and the left hip is less than 1.5, and the distance between the right wrist and the right hip is less than 1.5, and the distance between the left ankle and the right ankle is less than 1.5.

Judgment condition 2: continuous frame judgment, observation for 2 seconds——

Specifically, the relative distance between the left shoulder of the previous frame and the left shoulder of the next frame is in the range of [3,50], and the relative distance between the right shoulder of the previous frame and the right shoulder of the next frame is [3 ,50].

Judging condition 3: The wrist is under the shoulder.

For the arm-holding action, the definition of “left wrist and right elbow joint distance” or “right wrist-left elbow joint distance” is defined as L _c . Based on the standard distance and formula (3), the relative distance L _rc can be obtained. If the distance L _rc ＞2, it is regarded as an arm.

For leg cross movement, detect that the left ankle is on the right of the right ankle and the two knees are very close, the relative distance L _rx <1. Of course, it will also detect that the right ankle is on the left side of the left ankle and the knee is very close, the threshold is still 1.

There are many overlaps between hands to warm the neck and hands to breathe. To solve this problem, the present invention first detects the relative distance between the neck and the hands, if the relative distance is less than 2, and detects whether the wrists are under the nose; The distance between. If the distance is less than 3, it is judged to be a hand breathing; if the distance is greater than 3, it is judged to be holding the neck to warm up. Similarly, the above judgment thresholds are the empirical values obtained after the training test.

S7. Output posture and detection results. According to the corresponding relationship between the coordinate changes of the bone nodes and the judgment conditions and thresholds, the computer system automatically recognizes the posture and outputs the thermal tendency and thermal comfort level correspondingly.

In addition, the algorithm architecture also includes a posture detection and verification step S8. The specific operation is that the subject stands in the data collection area, mainly within the range of the data captured by the visual sensor. According to the 12 actions defined in the present invention, randomly Do the corresponding actions, the program automatically recognizes these actions, and outputs the thermal tendency and posture name. The detection results show that the algorithm can effectively identify related actions. It should be noted that the subjects invited in step S8 and the subjects involved in the questionnaire survey are two independent groups to ensure the objectivity of algorithm detection.

According to the posture defined in step S1, the subject makes random actions toward the computer vision device, and verifies the consistency of the output thermal tendency and thermal comfort level with the posture.

The application and implementation of the above algorithm architecture depend on the computer system. To this end, the computer architecture of the detection system includes:

Video acquisition and preprocessing unit, computer vision device to capture image data facing the subject, and used to preprocess and output interest domain pictures in the processor;

Generate the basic parameter matrix unit, which is used by the processor to capture the coordinates of the whole body skeleton node for each frame of interest domain picture input, and screen it frame by frame according to the confidence value;

The search area unit is used for the processor to search and delimit several regions related to the detection part one by one according to the action recognition requirements of the defined posture estimation. The delineation of the regions is based on the skeleton in the basic parameter matrix. The coordinates of the node;

The bone key-point locking unit is used for the processor to lock one or more bone node coordinates related to pose estimation in each area corresponding to each frame of interest domain picture;

The condition judgment unit is used for the processor to set the action recognition judgment conditions and thresholds corresponding to different posture estimations, and compare the change trend of the coordinates of each skeletal node in the previous and subsequent frames;

The output posture and detection result unit is used for the processor to recognize the posture according to the corresponding relationship between the coordinate change of the skeleton node and the judgment condition and threshold, and correspondingly output the thermal tendency and thermal comfort level of the detected human body.

In summary, the above embodiments combined with the detailed description of the figure, the application of the non-invasive human thermal comfort detection method and system of the present invention has outstanding substantive features and significant progress, which are specifically manifested in the following three outstanding aspects:

1. Convenient application. Compared with other physiological detection methods (including invasive and semi-invasive), it is easier to apply. People in the application scenario do not need to wear the sensor close to the body, or place it on glasses and other objects; non-invasive human thermal comfort detection is the direction of the future development of the central air-conditioning system. From an implementation perspective, it is also relatively convenient.

2. People-oriented, with the adoption of environmental testing methods as the mainstream, only 80% of the personnel are satisfied with a thermally comfortable environment. Ignoring the remaining 20% of the human experience and replacing the human experience with the ambient temperature, this detection method is more in line with the humanized performance requirements and realizes a people-oriented approach.

3. It saves energy and can provide effective feedback signals to the HVAC system in real time. Through the accumulation of data, it has a predictive function, thereby providing a thermal comfort environment suitable for indoor/car occupants. Optimizing on this basis can achieve all-round energy saving in small spaces, entire buildings, and entire communities.

The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the above specific embodiments. Those skilled in the art can make modifications or equivalent changes within the scope of the claims, and they should all be included in the protection scope of the present invention. within.

Claims (9)

1. A non-invasive human thermal comfort detection method based on attitude estimation, which is characterized in that it comprises the following steps:

   S1. Define several postures related to the thermal comfort of the human body, and verify the validity through questionnaire survey;

   S2: Video acquisition and preprocessing. The computer vision device captures and collects image data for subjects, and preprocesses and outputs pictures of interest domains;

   S3. Generate a basic parameter matrix, capture the coordinates of the whole body skeleton node for each frame of interest domain picture input, and filter frame by frame according to the confidence value;

   S4. Retrieval area, according to the action recognition requirements of the defined pose estimation, search and delimit several areas related to the detection part one by one in the interest domain pictures obtained by screening, and the delineation of the area is based on the coordinates of the skeleton node in the basic parameter matrix ；

   S5. Lock bone key points, and lock one or more bone node coordinates related to pose estimation in each area corresponding to each frame of interest domain picture;

   S6. Condition judgment, set the action recognition judgment conditions and thresholds corresponding to different posture estimations, and compare the change trends of the coordinates of each bone node in the previous and subsequent frames;

   S7. Output the posture and detection result, recognize the posture according to the corresponding relationship between the coordinate change of the bone node and the judgment condition and the threshold, and output the thermal tendency and thermal comfort level correspondingly.
2. The non-invasive human thermal comfort detection method based on posture estimation according to claim 1, wherein the posture related to thermal comfort of the human body in step S1 includes: wiping sweat and hand fan corresponding to the first thermal comfort level; T-shirt with shaking chest and head scratching in the second thermal comfort level, rolled sleeves corresponding to the third thermal comfort level, walking in the fourth thermal comfort level, shoulder shrinking corresponding to the fifth thermal comfort level, and corresponding to the sixth thermal comfort level Fold your arms, cross your legs, put your hands on your neck, and breathe or stom your feet at the seventh thermal comfort level.
3. The non-invasive human thermal comfort detection method based on posture estimation according to claim 1, wherein the image data in step S2 is 30 frames/sec continuous frames, and the preprocessing process is to extract the human posture frame by frame. Remove and enhance the image of the image, and output the interest domain image according to the area of the human body.
4. The non-invasive human thermal comfort detection method based on posture estimation according to claim 1, characterized in that: in step S3, by calling the OpenPose platform or bone capture algorithm, the whole body bone node information is directly obtained for each frame of interest domain image input. Coordinates and confidence values, output a parameter of i (x, y, ε), where the whole body bone nodes include nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, Right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear, left ear, numbered in the order of 0-17, and number 18 for the background of the picture, i represents the bone node and its corresponding number , X and y respectively represent the coordinate value of each skeletal node in the interest area picture coordinate system, ε represents the confidence value and set 0.5 as the screening threshold, adopt n frames of interest area pictures with ε≥0.5 and discard interest with ε<0.5 Domain picture.
5. The non-invasive human thermal comfort detection method based on posture estimation according to claim 1, characterized in that: according to the defined action recognition requirements of posture estimation, the area in step S4 area lock includes the forehead, the top of the head, and the left side of the head. Side, the right side of the head, the intersection area between the left side of the face and above the chest, the intersection area between the right side of the face and above the chest, the area between the neck and the wrists, the chest and abdomen areas, shoulders, elbows, and wrists The area between the hips, knees, and the ankles.
6. The non-invasive human thermal comfort detection method based on posture estimation according to claim 1, characterized in that: based on the several postures defined in step S1, the action corresponding to each posture is associated and matched with a part of the whole body skeleton node, and step The S5 dotfield locks the coordinates of a part of the bone nodes associated with a specific posture.
7. The non-invasive human thermal comfort detection method based on posture estimation according to claim 1, characterized in that: the judgment conditions in step S6 include the concentrated area where the displacement changes of more than one bone key points involved in posture estimation, relative distance, One or several combinations of the frequency and the range of distance change in the interval between the previous and next frames; the threshold value is an empirical value after the training test, including the associated relative distance threshold and the frequency threshold respectively.
8. The non-invasive human thermal comfort detection method based on posture estimation according to claim 1, characterized in that it further comprises a posture detection and verification step S8. The subject makes random actions towards the computer vision device according to the posture defined in step S1. And verify the consistency of the output thermal tendency and thermal comfort level with the posture.
9. A non-invasive human thermal comfort detection system based on posture estimation is realized by a computer and a number of pre-defined postures related to human thermal comfort, which is characterized by including:

   Video acquisition and preprocessing unit, computer vision device to capture image data facing the subject, and used to preprocess and output interest domain pictures in the processor;

   Generate the basic parameter matrix unit, which is used by the processor to capture the coordinates of the whole body skeleton node for each frame of interest domain picture input, and screen it frame by frame according to the confidence value;

   The search area unit is used for the processor to search and delimit several regions related to the detection part one by one according to the action recognition requirements of the defined posture estimation. The delineation of the regions is based on the skeleton in the basic parameter matrix. The coordinates of the node;

   The bone key-point locking unit is used for the processor to lock one or more bone node coordinates related to pose estimation in each area corresponding to each frame of interest domain picture;

   The condition judgment unit is used for the processor to set the action recognition judgment conditions and thresholds corresponding to different posture estimations, and compare the change trend of the coordinates of each skeletal node in the previous and subsequent frames;

   The output posture and detection result unit is used for the processor to recognize the posture according to the corresponding relationship between the coordinate change of the skeleton node and the judgment condition and threshold, and correspondingly output the thermal tendency and thermal comfort level of the detected human body.

Images