CN111166342A - Millimeter wave radar and camera fused fall detection device and detection method thereof - Google Patents
Millimeter wave radar and camera fused fall detection device and detection method thereof Download PDFInfo
- Publication number
- CN111166342A CN111166342A CN202010012523.3A CN202010012523A CN111166342A CN 111166342 A CN111166342 A CN 111166342A CN 202010012523 A CN202010012523 A CN 202010012523A CN 111166342 A CN111166342 A CN 111166342A
- Authority
- CN
- China
- Prior art keywords
- millimeter wave
- falling
- unit
- wave radar
- fall
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 75
- 238000010801 machine learning Methods 0.000 claims abstract description 28
- 238000012544 monitoring process Methods 0.000 claims abstract description 16
- 238000004891 communication Methods 0.000 claims abstract description 14
- 230000036544 posture Effects 0.000 claims abstract 4
- 238000012545 processing Methods 0.000 claims description 19
- 238000006073 displacement reaction Methods 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 13
- 230000004927 fusion Effects 0.000 claims description 11
- 230000000694 effects Effects 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 9
- 230000005484 gravity Effects 0.000 claims description 9
- 238000007781 pre-processing Methods 0.000 claims description 9
- 238000012360 testing method Methods 0.000 claims description 9
- 238000012795 verification Methods 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 5
- 230000017531 blood circulation Effects 0.000 claims description 4
- 230000003287 optical effect Effects 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 238000007635 classification algorithm Methods 0.000 claims description 2
- 230000000241 respiratory effect Effects 0.000 claims description 2
- 230000036387 respiratory rate Effects 0.000 claims description 2
- 210000000115 thoracic cavity Anatomy 0.000 claims description 2
- 238000013145 classification model Methods 0.000 claims 1
- 230000029058 respiratory gaseous exchange Effects 0.000 abstract description 3
- 238000000034 method Methods 0.000 description 8
- 230000007547 defect Effects 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 230000010354 integration Effects 0.000 description 2
- 238000007477 logistic regression Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 238000003062 neural network model Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000000038 chest Anatomy 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000036391 respiratory frequency Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/04—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
- G08B21/0407—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons based on behaviour analysis
- G08B21/043—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons based on behaviour analysis detecting an emergency event, e.g. a fall
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0002—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/0205—Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
- A61B5/02055—Simultaneously evaluating both cardiovascular condition and temperature
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1116—Determining posture transitions
- A61B5/1117—Fall detection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/103—Detecting, measuring or recording devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
- A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb
- A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique
- A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor, mobility of a limb using a particular sensing technique using image analysis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/74—Details of notification to user or communication with user or patient ; user input means
- A61B5/742—Details of notification to user or communication with user or patient ; user input means using visual displays
- A61B5/743—Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
- G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
- G01S13/867—Combination of radar systems with cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/04—Alarms for ensuring the safety of persons responsive to non-activity, e.g. of elderly persons
- G08B21/0438—Sensor means for detecting
- G08B21/0492—Sensor dual technology, i.e. two or more technologies collaborate to extract unsafe condition, e.g. video tracking and RFID tracking
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Animal Behavior & Ethology (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Heart & Thoracic Surgery (AREA)
- Biomedical Technology (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Public Health (AREA)
- Surgery (AREA)
- Theoretical Computer Science (AREA)
- Physiology (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Cardiology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Software Systems (AREA)
- Dentistry (AREA)
- Radiology & Medical Imaging (AREA)
- Psychiatry (AREA)
- Gerontology & Geriatric Medicine (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Computer Networks & Wireless Communication (AREA)
- Social Psychology (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
Abstract
The invention provides a millimeter wave radar and camera fused fall detection device and a detection method thereof, wherein a fall detection sensor acquires physiological signs and body postures, and an application server is responsible for providing access service for the fall detection sensor and pushing the field situation and sign information of a fall event to a mobile terminal monitoring module in real time; the fall detection sensor comprises a visible light image sensing unit, a millimeter wave radar unit, a machine learning inference unit and a communication unit; the visible light image unit and the millimeter wave radar unit are respectively connected with the machine learning inference unit, and the machine learning inference unit is connected with the communication unit. The invention can self-adapt to the ambient light to judge the falling state and can normally work under weak light and no light. Monitoring physiological signs and body postures of a person under guardianship such as respiration and heart rate when the person does not fall down or falls down, and pushing pictures and physiological signs to a terminal when the person falls down.
Description
Technical Field
The invention belongs to the technical field of digital signal processor application, and particularly relates to a millimeter wave radar and camera fused fall detection device and a detection method thereof.
Background
Fall detection is divided into two modes, namely contact detection and non-contact detection. The contact detection generally adopts to wear wearable equipment, judges the state of tumbleing through acceleration sensor motion characteristic, and the shortcoming of contact equipment is that need wear relevant equipment and just can detect, needs regularly to charge, and is poor to users' ease of use such as old man. The non-contact detection mainly judges the falling state through modes of electromagnetic waves, optical sensors and the like and a related algorithm, has the advantages of no need of wearing equipment, no binding feeling and suitability for monitoring falling of people in a specific area.
One of the more popular schemes at present is that a visible light image sensor is used for analyzing the movement posture of a person through collecting regional pictures and a machine learning algorithm, so that whether the person falls or not can be judged. The method has the advantages that the method has high accuracy by using a deep learning algorithm and a visualization function, and the fallen scene pictures and videos can be pushed to the application programs of the relatives' mobile phones.
The disadvantage is that the visible light image sensor is relatively easily influenced by the ambient light. The imaging effect is poor in a weak light environment, so that the personnel detection rate is low, and the corresponding falling recognition rate is low. Even in a dark environment, it is completely inoperable.
By adopting the infrared image sensor, the defects of the visible light sensor can be overcome, but the infrared light imaging effect is poor, and the recognition rate is lower than that of visible light. More importantly, the infrared image sensor is usually very high in price and low in practical value.
In addition, the millimeter wave radar sensor can detect the Doppler displacement of a human body, and the falling state of the person can be judged by detecting the position change of the human body in the vertical direction. Its advantages are low cost and high effect on normal operation in no light. However, the millimeter wave radar sensor scheme has the defects of high technical difficulty and incapability of directly observing the field condition.
Disclosure of Invention
Aiming at the technical problems, the invention provides a millimeter wave radar and camera fused fall detection device and a detection method thereof, which solve the problems that the visible light sensor is low in recognition rate in a weak light environment and cannot work in a no light environment; the problems of poor imaging effect, low recognition rate, high cost and the like of an infrared image sensor are solved; the defects that visual checking cannot be performed by adopting a millimeter wave radar and the like are overcome.
The specific technical scheme is as follows:
a millimeter wave radar and camera integrated fall detection device comprises a fall detection sensor, an application server and a mobile terminal monitoring module;
the fall detection sensor collects physiological signs, and the application server is responsible for providing access service for the fall detection sensor and pushing the field situation and sign information of a fall event to the mobile terminal monitoring module in real time.
The fall detection sensor comprises a visible light image unit, a millimeter wave radar unit, a machine learning inference unit and a communication unit;
the visible light image unit and the millimeter wave radar unit are respectively connected with the machine learning inference unit, and the machine learning inference unit is connected with the communication unit.
Preferably, the visible light imaging unit is an integrated SOC chip. Or may be a discrete chip.
Furthermore, the visible light image unit comprises a visible light image sensor and an image signal processing module, and is responsible for visible light image acquisition and preprocessing;
the visible light image sensor comprises an optical lens, a visible light image sensor and a sensor controller;
and the image signal processing module is responsible for image signal preprocessing.
Preferably, the millimeter wave radar unit is an integrated SOC chip. And a radio frequency front-end chip and a digital back-end chip which are separated can be adopted, so that the cost is reduced.
Furthermore, the millimeter wave radar unit comprises a millimeter wave sensor and a digital signal processing unit, and completes the functions of millimeter wave modulation, demodulation, IQ signal processing and sign signal calculation.
The millimeter wave sensor comprises a millimeter wave antenna, a radio frequency processing front end and a digital-to-analog signal conversion module.
The digital signal processing unit comprises a digital signal preprocessing module and a physical sign monitoring module.
The machine learning inference unit comprises a processor for machine learning inference and is mainly responsible for posture detection and falling fusion detection of personnel.
The communication unit is a communication module for accessing a network.
In addition, as an optimal design, the detection device reserves a Flash memory chip and an SD card slot for storing key frame compressed pictures in a certain period, and adopts a cyclic replacement mode according to time when the storage space is full.
The detection method of the fall detection device with the integration of the millimeter wave radar and the camera comprises the following steps:
the millimeter wave thunder unit collects radio frequency signals and calculates parameters of a human body:
vertical displacement mm _ dy
Horizontal displacement mm _ dx
Vertical displacement speed mm _ dvy
The horizontal direction displacement speed mm _ dvx,
respiratory induced thoracic displacement mm _ db
Respiratory rate mm _ br
Heart rate mm _ hr
(2) Acquiring by a visible light image unit:
height img _ dp of human body gravity center
Arm force gravity center height img _ dh
Height of center of gravity img _ dl of leg
Detecting the orientation img _ theta of the human body
Calculating the heart rate img _ br according to the blood flow change of the human face;
(3) taking the parameters collected in (1) and (2) as fall detection input, and recording as a vector X ═ mm _ dy, mm _ dx, mm _ dvy, mm _ dvx, mm _ db, mm _ br, mm _ hr, img _ dp, img _ dh, img _ theta, img _ dl and img _ br); the problem of detecting falling is converted into a two-classification problem to be processed.
Calculating corresponding output of an X input model Y { -F (X), wherein F is a classifier model learned through historical data, the value range of Y is { -1, 1}, the detected fall is represented as 1, and the undetected fall is represented as-1; calculating an intermediate value of the X input model Y ═ F (X), and normalizing to obtain a falling probability P;
the invention adopts a machine learning fusion algorithm, and the realization method is as follows:
step 1, detecting physical signs of an experimenter when falling by using a falling detection sensor and sending the physical signs to an acquisition computer; the system is distributed in the light environment of strong light, weak light and no light, the acquisition sensor acquires a parameter X, the real falling state is recorded, the falling state is recorded as 1, and the non-falling state is recorded as-1;
step 2, repeating the step 1, and collecting experimental data as a training, verifying and testing data set;
dividing data into training set, verification set and test set according to a certain proportion
The binary classification algorithm is adopted as a model, and the model is not limited to a logistic regression model, a support vector machine model, a random gradient descent model, a perception machine model, a neural network model and the like, the binary model is trained by using training set data, the training effect is verified by using verification set data, and the actual effect of the model is observed by using test set data;
step 3, adjusting the parameters of the hyper-parameter optimization model;
and 4, transplanting the trained model to a falling detection machine learning inference unit to serve as a falling decision model.
The invention provides a millimeter wave radar and camera fused fall detection device and a detection method thereof, compared with the prior art, the device has the following beneficial effects:
(1) the millimeter wave radar and camera integrated falling detection device and the detection method thereof provided by the invention can be used for self-adaptively judging the falling state by ambient light.
(2) The device and the method for detecting the falling of the integrated millimeter wave radar and camera provided by the invention can work normally under weak light and no light.
(3) The millimeter wave radar and camera integrated falling detection device and the detection method thereof provided by the invention can detect physiological signs such as respiration and heart rate when people do not fall down or fall down.
(4) The millimeter wave radar and camera integrated falling detection device and the detection method thereof provided by the invention can push pictures and physiological signs to the relative mobile phone when falling.
(5) The millimeter wave radar and camera integrated fall detection device and the detection method thereof provided by the invention can be used for viewing pictures, videos and physiological sign information of specified equipment through an application program or a client with an authentication function on mobile equipment such as a mobile phone.
(6) The device and the method for detecting the falling of the device with the integration of the millimeter wave radar and the camera can automatically dial a set calling phone, such as a family phone or an emergency phone such as 120 and the like, when the device finds that the falling occurs after the setting.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a schematic structural view of the present invention;
fig. 2 is a schematic view of the usage state of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
In the description of the embodiments of the present invention, it should be further noted that, unless otherwise explicitly stated or limited, the terms "disposed," "mounted," "connected," and "connected" are to be construed broadly and may include, for example, a fixed connection, a detachable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
The millimeter wave radar and camera fused fall detection device provided by the invention combines the defects and cost consideration of the prior art, and adopts a scheme of combining a visible light image sensor and a millimeter wave radar sensor (not limited to 24GHz and 77GHz) as a fall detection sensor.
The system comprises a fall detection sensor, an application server program and a monitoring application program, wherein the fall detection sensor, the application server program and the monitoring application program form a complete millimeter wave radar and camera fusion fall detection device, the fall detection sensor acquires physiological signs, and the application server program is responsible for providing access service for the fall detection sensor and pushing the scene situation and sign information of a fall event to the monitoring application program in real time.
The specific structural schematic diagram is shown in fig. 1.
Specifically, the fall detection sensor is composed of a visible light image unit, a millimeter wave radar unit, a machine learning inference unit and a communication unit.
The visible light image unit and the millimeter wave radar unit are respectively connected with the machine learning inference unit, and the machine learning inference unit is connected with the communication unit;
wherein the visible light image unit may be an integrated SOC chip. The visible light image unit comprises a visible light image sensor and an image signal processing module and is responsible for visible light image acquisition and preprocessing. The visible light image sensor includes an optical lens, a visible light image sensor, and a sensor controller. The image signal processing module is responsible for image signal preprocessing.
Wherein the millimeter wave radar unit may be an integrated SOC chip. The millimeter wave radar unit comprises a millimeter wave sensor and a digital signal processing unit, and completes functions of millimeter wave modulation, demodulation, IQ signal processing, sign signal calculation and the like. The millimeter wave sensor comprises a millimeter wave antenna, a radio frequency processing front end, a digital-to-analog signal conversion module and the like. The digital signal processing unit comprises modules of digital signal preprocessing, physical sign monitoring and the like.
Preferably, the machine learning inference unit includes an edge processor and a cloud computing platform for machine learning inference, and is not limited to processors of computing architectures such as X86, ARM, DSP, NPU, GPU, and the like, and is mainly responsible for posture detection and fall fusion detection of people.
The communication unit is not limited to Zigbee, bluetooth, WIFI, GSM, WCDMA, LTE, 5G, NB-IOT, etc. and may be used to access the communication module of the network.
Besides, the detection device reserves a Flash memory chip and an SD card slot for storing key frame compressed pictures in a certain period, and adopts a cyclic replacement mode according to time when the storage space is full.
The working state of the millimeter wave radar and camera integrated fall detection device provided by the invention is shown in fig. 2, and the specific flow is as follows:
the millimeter wave thunder unit calculates and collects vertical displacement mm _ dy, horizontal displacement mm _ dx, vertical displacement speed mm _ dvy and horizontal displacement speed mm _ dvx according to the Doppler principle; the phase information is separated and filtered by extracting the phase information in the barrier area and the frequency range, the phase information with the frequency of 0.1-0.6Hz is filtered to calculate the chest displacement mm _ db and the respiratory frequency mm _ br caused by respiration, and the phase information with the frequency of 1.6-4.0Hz is filtered to be used for extracting the heart rate mm _ hr.
The method comprises the steps of obtaining key feature points of a body, arms and legs through a visible light camera posture detection algorithm which is not limited to a machine learning method for extracting features and a neural network, calculating the height of a person, the gravity center of the person, the position of the arms and the position of the legs through a trigonometric relation, and estimating to obtain the height img _ dp of the gravity center of the body, the height img _ dh of the force of the arms and the height img _ dl of the gravity center of the legs.
And detecting the personnel existing in the image through human body target detection, and calculating the orientation img _ theta of the personnel according to the pixel distribution condition.
The heart rate img _ br calculated by the human face blood flow change can be detected through the human face blood flow change caused by the pulse;
the sensor output X is collected and the corresponding fatigue state Y, Y is an instance of Y and the corresponding probability P, P is an instance of P. And training a machine learning fusion algorithm by the acquired data set to obtain a model F.
The machine learning model F trained in advance is transplanted to the reasoning unit, and the newly acquired sensor data is input into the reasoning unit, so that the corresponding falling state and falling probability can be obtained.
Specifically, a machine learning fusion algorithm implementation method is adopted, and the details are as follows:
step 1, detecting physical signs of an experimenter when falling by using a falling detection sensor and sending the physical signs to an acquisition computer; the system is distributed in the light environment of strong light, weak light and no light, the acquisition sensor acquires a parameter X, the real falling state is recorded, the falling state is recorded as 1, and the non-falling state is recorded as-1;
step 2, repeating the step 1, and collecting experimental data as a training, verifying and testing data set;
step 3, dividing the data into a training set, a verification set and a test set according to a certain proportion;
the binary algorithm is adopted as a model, and the model is not limited to a logistic regression model, a support vector machine model, a random gradient descent model, a perception machine model, a neural network model and the like, the network is trained by utilizing training set data, the training effect is verified by utilizing verification set data, and the actual effect of the model is observed by utilizing test set data;
step 4, adjusting the parameters of the hyper-parametric optimization model by methods such as random parameter search, Bayesian optimization and the like;
and 5, transplanting the trained model to a falling detection machine learning inference unit to serve as a falling decision model.
The system outputs a falling state Y, a falling probability P and a sensor acquisition parameter X to be updated to a server in real time; when the falling state is detected, the falling state Y, the falling probability P, the sensor acquisition parameter X and a plurality of scene pictures during falling are uploaded to the server, and push messages are generated and pushed to the monitoring end application program.
In the description herein, references to the description of "one embodiment," "an example," "a specific example" or the like are intended to mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The foregoing is merely exemplary and illustrative of the present invention and various modifications, additions and substitutions may be made by those skilled in the art to the specific embodiments described without departing from the scope of the invention as defined in the following claims.
Claims (10)
1. A millimeter wave radar and camera integrated fall detection device is characterized by comprising a fall detection sensor, an application server and a mobile terminal monitoring module;
the system comprises a falling detection sensor, an application server and a mobile terminal monitoring module, wherein the falling detection sensor acquires physiological signs and human body postures, and the application server is responsible for providing access service for the falling detection sensor and pushing the field condition and sign information of a falling event to the mobile terminal monitoring module in real time;
the fall detection sensor unit comprises a visible light image unit, a millimeter wave radar unit, a machine learning inference unit and a communication unit;
the visible light image unit and the millimeter wave radar unit are respectively connected with the machine learning inference unit, and the machine learning inference unit is connected with the communication unit.
2. The millimeter wave radar and camera fused fall detection device according to claim 1, wherein the visible light image unit comprises an integrated SOC chip, a discrete rf front end and a digital back end combined scheme.
3. The device for detecting falling through fusion of a millimeter wave radar and a camera as claimed in claim 1, wherein the visible light image unit comprises a visible light image sensor and an image signal processing module, and is responsible for visible light image acquisition and preprocessing;
the visible light image sensor comprises an optical lens, a visible light image sensor and a sensor controller;
and the image signal processing module is responsible for image signal preprocessing.
4. The device of claim 1, wherein the millimeter wave radar unit is an integrated SOC chip or a discrete RF front-end chip and a discrete digital back-end chip.
5. The device for detecting the falling through the fusion of the millimeter wave radar and the camera according to claim 1, wherein the millimeter wave radar unit comprises a millimeter wave sensor and a digital signal processing unit, and the functions of millimeter wave modulation, demodulation, IQ signal processing and sign signal calculation are completed;
the millimeter wave sensor comprises a millimeter wave antenna, a radio frequency processing front end and a digital-to-analog signal conversion module;
the digital signal processing unit comprises a digital signal preprocessing module and a physical sign monitoring module.
6. The device for detecting the integrated fall of the millimeter wave radar and the camera as claimed in claim 1, wherein the machine learning inference unit comprises an edge processor and a cloud inference server for machine learning inference, and is mainly responsible for posture detection and fall integrated detection of personnel.
7. The device for detecting fall through fusion of a millimeter wave radar and a camera as claimed in claim 1, wherein the communication unit is a communication module for accessing a network.
8. The device for detecting the falling through the fusion of the millimeter wave radar and the camera as claimed in claim 1, further comprising a reserved Flash memory chip and an SD card slot for storing key frame compressed pictures in a certain period, wherein a cyclic replacement mode is adopted according to time when the storage space is full.
9. The detection method of the millimeter wave radar and camera fused fall detection device according to any one of claims 1 to 8, comprising the following steps:
(1) the millimeter wave thunder unit collects radio frequency signals and calculates parameters of a human body:
vertical displacement mm _ dy
Horizontal displacement mm _ dx
Vertical displacement speed mm _ dvy
The horizontal direction displacement speed mm _ dvx,
respiratory induced thoracic displacement mm _ db
Respiratory rate mm _ br
Heart rate mm _ hr
(2) Acquiring by a visible light image unit:
height img _ dp of human body gravity center
Arm force gravity center height img _ dh
Height of center of gravity img _ dl of leg
Detecting the orientation img _ theta of the human body
Calculating the heart rate img _ br according to the blood flow change of the human face;
(3) taking the parameters collected in (1) and (2) as fall detection input, and recording as a vector X ═ mm _ dy, mm _ dx, mm _ dvy, mm _ dvx, mm _ db, mm _ br, mm _ hr, img _ dp, img _ dh, img _ theta, img _ dl and img _ br); the problem of detecting falling is converted into a two-classification problem for processing;
calculating corresponding output of an X input model Y { -F (X), wherein F is a classifier model learned through historical data, the value range of Y is { -1, 1}, the detected fall is represented as 1, and the undetected fall is represented as-1; calculating an intermediate value of the X input model Y ═ F (X), and normalizing to obtain a falling probability P;
(4) the system outputs a falling state Y, a falling probability P and a sensor acquisition parameter X to be updated to a server in real time; when the falling state is detected, the falling state Y, the falling probability P, the sensor acquisition parameter X and a plurality of scene pictures during falling are uploaded to the server, and push messages are generated and pushed to the monitoring end application program.
10. The detection method of the millimeter wave radar and camera fused fall detection device according to claim 9, wherein a machine learning fusion algorithm is adopted, and the detection method specifically comprises the following substeps:
step 1, detecting physical signs of an experimenter when falling by using a falling detection sensor and sending the physical signs to an acquisition computer; the system is distributed in the light environment of strong light, weak light and no light, the acquisition sensor acquires a parameter X, the real falling state is recorded, the falling state is recorded as 1, and the non-falling state is recorded as-1;
step 2, repeating the step 1, and collecting experimental data as a training, verifying and testing data set;
dividing data into training set, verification set and test set according to a certain proportion
A binary classification algorithm is adopted as a model, a binary classification model is trained by using training set data, a training effect is verified by using verification set data, and an actual effect of the model is observed by using test set data;
step 3, adjusting the parameters of the hyper-parameter optimization model;
and 4, transplanting the trained model to a falling detection machine learning inference unit to serve as a falling decision model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010012523.3A CN111166342B (en) | 2020-01-07 | 2020-01-07 | Millimeter wave radar and camera fused fall detection device and detection method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010012523.3A CN111166342B (en) | 2020-01-07 | 2020-01-07 | Millimeter wave radar and camera fused fall detection device and detection method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111166342A true CN111166342A (en) | 2020-05-19 |
| CN111166342B CN111166342B (en) | 2022-10-14 |
Family
ID=70619860
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010012523.3A Active CN111166342B (en) | 2020-01-07 | 2020-01-07 | Millimeter wave radar and camera fused fall detection device and detection method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111166342B (en) |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111568437A (en) * | 2020-06-01 | 2020-08-25 | 浙江大学 | Non-contact type bed leaving real-time monitoring method |
| CN112346055A (en) * | 2020-10-23 | 2021-02-09 | 无锡威孚高科技集团股份有限公司 | Fall detection method and device based on millimeter wave radar and millimeter wave radar equipment |
| CN112782681A (en) * | 2020-12-31 | 2021-05-11 | 杭州电子科技大学 | Indoor positioning and falling detection system and method based on millimeter waves and Internet of things |
| CN112784890A (en) * | 2021-01-14 | 2021-05-11 | 泰康保险集团股份有限公司 | Human body falling detection method and system by means of multi-sensor fusion |
| CN112859187A (en) * | 2021-01-06 | 2021-05-28 | 路晟（上海）科技有限公司 | Method, device, equipment and system for recognizing posture of detected object |
| CN113030948A (en) * | 2021-03-03 | 2021-06-25 | 上海大学 | Fall detection system and method based on target following millimeter wave radar |
| CN113116342A (en) * | 2021-04-26 | 2021-07-16 | 上海交通大学 | Life signal monitoring method based on millimeter wave radar |
| CN113425289A (en) * | 2021-05-24 | 2021-09-24 | 上海可深信息科技有限公司 | Method for identifying person falling |
| CN113447905A (en) * | 2021-06-29 | 2021-09-28 | 西安电子科技大学 | Double-millimeter-wave radar human body falling detection device and detection method |
| CN113740848A (en) * | 2021-09-30 | 2021-12-03 | 中科芯集成电路有限公司 | Tracking type fall detection device for solitary old people based on millimeter wave radar |
| CN114005246A (en) * | 2021-01-29 | 2022-02-01 | 江苏中科西北星信息科技有限公司 | Old man falling detection method and device based on frequency modulation continuous wave millimeter wave radar |
| CN114442079A (en) * | 2022-01-14 | 2022-05-06 | 北京清雷科技有限公司 | Target object falling detection method and device |
| CN114511916A (en) * | 2022-04-19 | 2022-05-17 | 亿慧云智能科技（深圳）股份有限公司 | Face recognition-based terminal equipment control method and device and terminal equipment |
| CN114720977A (en) * | 2022-06-10 | 2022-07-08 | 长沙莫之比智能科技有限公司 | Object falling detection method and device, millimeter wave radar and storage medium |
| WO2022168091A1 (en) * | 2021-02-03 | 2022-08-11 | Elbit Systems C4I and Cyber Ltd. | System and method for remote multimodal sensing of a measurement volume |
| CN115359624A (en) * | 2022-10-24 | 2022-11-18 | 中科芯集成电路有限公司 | Millimeter wave radar fall detection algorithm based on state quantity analysis and sign detection |
| CN116319899A (en) * | 2023-05-18 | 2023-06-23 | 三亚明熹一品信息技术服务有限公司 | System for online monitoring activity state of human body |
| CN116631155A (en) * | 2023-06-01 | 2023-08-22 | 深圳市震有智联科技有限公司 | Old people falling identification method and automatic help calling system |
| CN116687394A (en) * | 2023-08-04 | 2023-09-05 | 亿慧云智能科技（深圳）股份有限公司 | Tumble detection method, device, equipment and storage medium based on millimeter wave radar |
| US12027028B2 (en) | 2020-06-03 | 2024-07-02 | Essence Smartcare Ltd. | Controlling frame rate of active reflected wave detector |
| CN118430070A (en) * | 2024-05-27 | 2024-08-02 | 深圳市国关智能技术有限公司 | Human behavior recognition and data acquisition system based on artificial intelligence |
| US12087144B2 (en) | 2020-05-21 | 2024-09-10 | Essence Smartcare Ltd. | Device and method for determining a status of a person |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107920737A (en) * | 2015-05-31 | 2018-04-17 | Sens4保护公司 | The remote supervision system of mankind's activity |
| CN108968970A (en) * | 2018-05-24 | 2018-12-11 | 厦门精益远达智能科技有限公司 | A kind of method, apparatus and radar system that Doppler's millimetre-wave radar detection human body is fallen |
| KR102038081B1 (en) * | 2018-10-16 | 2019-10-29 | 주식회사 젠다카디언 | Device for detecting fall and rise |
| KR20190134911A (en) * | 2018-05-11 | 2019-12-05 | 삼성전자주식회사 | Electronic device and method for controlling the electronic device thereof |
-
2020
- 2020-01-07 CN CN202010012523.3A patent/CN111166342B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107920737A (en) * | 2015-05-31 | 2018-04-17 | Sens4保护公司 | The remote supervision system of mankind's activity |
| KR20190134911A (en) * | 2018-05-11 | 2019-12-05 | 삼성전자주식회사 | Electronic device and method for controlling the electronic device thereof |
| CN108968970A (en) * | 2018-05-24 | 2018-12-11 | 厦门精益远达智能科技有限公司 | A kind of method, apparatus and radar system that Doppler's millimetre-wave radar detection human body is fallen |
| KR102038081B1 (en) * | 2018-10-16 | 2019-10-29 | 주식회사 젠다카디언 | Device for detecting fall and rise |
Non-Patent Citations (1)
| Title |
|---|
| FENG JIN等: "Multiple Patients Behavior Detection in Real-time using mmWave Radar and Deep CNNs", 《2019 IEEE RADAR CONFERENCE (RADARCONF)》 * |
Cited By (34)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US12087144B2 (en) | 2020-05-21 | 2024-09-10 | Essence Smartcare Ltd. | Device and method for determining a status of a person |
| CN111568437B (en) * | 2020-06-01 | 2021-07-09 | 浙江大学 | Non-contact type bed leaving real-time monitoring method |
| CN111568437A (en) * | 2020-06-01 | 2020-08-25 | 浙江大学 | Non-contact type bed leaving real-time monitoring method |
| US12027028B2 (en) | 2020-06-03 | 2024-07-02 | Essence Smartcare Ltd. | Controlling frame rate of active reflected wave detector |
| CN112346055A (en) * | 2020-10-23 | 2021-02-09 | 无锡威孚高科技集团股份有限公司 | Fall detection method and device based on millimeter wave radar and millimeter wave radar equipment |
| CN112346055B (en) * | 2020-10-23 | 2024-04-16 | 无锡威孚高科技集团股份有限公司 | Fall detection method and device based on millimeter wave radar and millimeter wave radar equipment |
| CN112782681B (en) * | 2020-12-31 | 2024-05-24 | 杭州电子科技大学 | Indoor positioning and falling detection system and method based on millimeter wave and Internet of things |
| CN112782681A (en) * | 2020-12-31 | 2021-05-11 | 杭州电子科技大学 | Indoor positioning and falling detection system and method based on millimeter waves and Internet of things |
| CN112859187A (en) * | 2021-01-06 | 2021-05-28 | 路晟（上海）科技有限公司 | Method, device, equipment and system for recognizing posture of detected object |
| CN112859187B (en) * | 2021-01-06 | 2022-11-08 | 路晟（上海）科技有限公司 | Method, device, equipment and system for recognizing posture of detected object |
| CN112784890A (en) * | 2021-01-14 | 2021-05-11 | 泰康保险集团股份有限公司 | Human body falling detection method and system by means of multi-sensor fusion |
| CN112784890B (en) * | 2021-01-14 | 2024-01-26 | 泰康保险集团股份有限公司 | Human body fall detection method and system by utilizing multi-sensor fusion |
| CN114005246A (en) * | 2021-01-29 | 2022-02-01 | 江苏中科西北星信息科技有限公司 | Old man falling detection method and device based on frequency modulation continuous wave millimeter wave radar |
| CN114005246B (en) * | 2021-01-29 | 2024-01-30 | 江苏中科西北星信息科技有限公司 | Fall detection method and device for old people based on frequency modulation continuous wave millimeter wave radar |
| WO2022168091A1 (en) * | 2021-02-03 | 2022-08-11 | Elbit Systems C4I and Cyber Ltd. | System and method for remote multimodal sensing of a measurement volume |
| IL280630A (en) * | 2021-02-03 | 2022-09-01 | Elbit Systems C4I And Cyber Ltd | System and method for remote multimodal sensing of a measurement volume |
| CN113030948A (en) * | 2021-03-03 | 2021-06-25 | 上海大学 | Fall detection system and method based on target following millimeter wave radar |
| CN113116342A (en) * | 2021-04-26 | 2021-07-16 | 上海交通大学 | Life signal monitoring method based on millimeter wave radar |
| CN113425289A (en) * | 2021-05-24 | 2021-09-24 | 上海可深信息科技有限公司 | Method for identifying person falling |
| CN113447905A (en) * | 2021-06-29 | 2021-09-28 | 西安电子科技大学 | Double-millimeter-wave radar human body falling detection device and detection method |
| CN113447905B (en) * | 2021-06-29 | 2024-04-19 | 西安电子科技大学 | Double millimeter wave radar human body fall detection device and detection method |
| CN113740848A (en) * | 2021-09-30 | 2021-12-03 | 中科芯集成电路有限公司 | Tracking type fall detection device for solitary old people based on millimeter wave radar |
| CN113740848B (en) * | 2021-09-30 | 2023-11-17 | 中科芯集成电路有限公司 | Solitary old person tracking type falling detection device based on millimeter wave radar |
| CN114442079A (en) * | 2022-01-14 | 2022-05-06 | 北京清雷科技有限公司 | Target object falling detection method and device |
| CN114511916B (en) * | 2022-04-19 | 2022-06-17 | 亿慧云智能科技（深圳）股份有限公司 | Face recognition-based terminal equipment control method and device and terminal equipment |
| CN114511916A (en) * | 2022-04-19 | 2022-05-17 | 亿慧云智能科技（深圳）股份有限公司 | Face recognition-based terminal equipment control method and device and terminal equipment |
| CN114720977A (en) * | 2022-06-10 | 2022-07-08 | 长沙莫之比智能科技有限公司 | Object falling detection method and device, millimeter wave radar and storage medium |
| CN115359624A (en) * | 2022-10-24 | 2022-11-18 | 中科芯集成电路有限公司 | Millimeter wave radar fall detection algorithm based on state quantity analysis and sign detection |
| CN116319899A (en) * | 2023-05-18 | 2023-06-23 | 三亚明熹一品信息技术服务有限公司 | System for online monitoring activity state of human body |
| CN116631155A (en) * | 2023-06-01 | 2023-08-22 | 深圳市震有智联科技有限公司 | Old people falling identification method and automatic help calling system |
| CN116631155B (en) * | 2023-06-01 | 2024-07-05 | 深圳市震有智联科技有限公司 | Old people falling identification method and automatic help calling system |
| CN116687394B (en) * | 2023-08-04 | 2023-10-20 | 亿慧云智能科技（深圳）股份有限公司 | Tumble detection method, device, equipment and storage medium based on millimeter wave radar |
| CN116687394A (en) * | 2023-08-04 | 2023-09-05 | 亿慧云智能科技（深圳）股份有限公司 | Tumble detection method, device, equipment and storage medium based on millimeter wave radar |
| CN118430070A (en) * | 2024-05-27 | 2024-08-02 | 深圳市国关智能技术有限公司 | Human behavior recognition and data acquisition system based on artificial intelligence |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111166342B (en) | 2022-10-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111166342B (en) | Millimeter wave radar and camera fused fall detection device and detection method thereof | |
| US10506990B2 (en) | Devices and methods for fall detection based on phase segmentation | |
| US20180300538A1 (en) | Image processing system, image processing apparatus, image processing method, and image processing program | |
| CN102982341B (en) | Self-intended crowd density estimation method for camera capable of straddling | |
| CN105556539A (en) | Detection devices and methods for detecting regions of interest | |
| CN109171734A (en) | Human body behavioural analysis cloud management system based on Fusion | |
| WO2020044180A2 (en) | Method and system for gaze estimation | |
| Jansi et al. | Detection of fall for the elderly in an indoor environment using a tri-axial accelerometer and Kinect depth data | |
| Li et al. | Collaborative fall detection using smart phone and Kinect | |
| Soni et al. | Automated fall detection from a camera using support vector machine | |
| CN113545774A (en) | Indoor and outdoor fall detection system based on depth measurement device and wearable device | |
| CN103961108B (en) | To fall method for detecting | |
| Zhu et al. | Falling motion detection algorithm based on deep learning | |
| Merrouche et al. | Fall detection based on shape deformation | |
| De et al. | Fall detection method based on spatio-temporal feature fusion using combined two-channel classification | |
| Roshini et al. | An Enhanced Posture Prediction‐Bayesian Network Algorithm for Sleep Posture Recognition in Wireless Body Area Networks | |
| De et al. | Fall detection approach based on combined two-channel body activity classification for innovative indoor environment | |
| CN112395920B (en) | Gesture recognition device and method based on radar and electronic equipment | |
| Nguyen et al. | Human gait symmetry assessment using a depth camera and mirrors | |
| CN114202797A (en) | Behavior recognition method, behavior recognition device and storage medium | |
| US20220027726A1 (en) | Pre-processing device, pre-processing method, and pre-processing program | |
| CN110555394A (en) | Fall risk assessment method based on human body shape characteristics | |
| Barelle et al. | The BIOTELEKINESY home care service: A holistic concept to prevent fall among the elderly based on ICT and computer vision | |
| Wu et al. | Unobtrusive monitoring of sedentary behaviors with fusion of bluetooth and ballistocardiogram signals | |
| CN111861275B (en) | Household work mode identification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20240918 Address after: Room 1003, 10th floor, unit 2, building 7, No. 399, West Fucheng Avenue, high tech Zone, Chengdu, Sichuan 610000 Patentee after: Sichuan Yuran Electronic Technology Co.,Ltd. Country or region after: China Address before: Room 425, Optoelectronic Building, University of Electronic Science and Technology of China, No. 4, Section 2, Jianshe North Road, Chenghua District, Chengdu, Sichuan 610054 Patentee before: Sichuan Yuran Zhihui Technology Co.,Ltd. Country or region before: China |