TWI820784B - A fall and posture identifying method with safety caring and high identification handling - Google Patents

Abstract

This invention relating to a fall and posture identifying method with safety caring and high identification handling, is for a fall and posture identify identification system. This includes an image capture unit, an image processing unit and a fall identification unit. Firstly, the image capturing unit captures a body images of care recipients to generate a body image data corresponding to each care recipient. Then the image capture unit the image processing unit receives the body image data and preprocesses the body image data for image quality and noise. Finally, the fall identification unit receives the body image data after image preprocessing, and judges whether the care recipient falls, and to issued instant notification message if the care recipient falls.

Description

A fall and posture recognition method with safe care and high privacy processing

The present invention relates to a fall and posture recognition method with safe care and high privacy processing, especially a fall, posture, behavior, etc. identification method with safe care and high privacy processing that uses smart care and can protect the privacy of the person being cared for.

With the advent of the aging era, safety care issues such as safe care for the elderly at home are constantly emerging. In particular, reporting of high-risk falls is a need to minimize injuries and allow the care recipients to receive proper care in real time. However, in the past, The nursing method requires huge medical labor costs. In order to reduce labor costs, the most common method is to monitor in real time through a camera. However, during the long-term care process, the privacy of the person being cared for is not easy to be protected.

The current existing technology is to let the person being cared for wear a wearable smart detection bracelet, and use the internal gyroscope to analyze and determine whether the person being cared for has fallen, and send a notification when the person falls. However, this type of The detection method can easily cause misjudgments due to the daily activities and body movements or behaviors of the care recipient. On the other hand, wearing the sensor can also easily cause physical discomfort, and the care recipient may forget to carry or wear it.

In addition, referring to Figures 22 and 23 together, the commonly known US patent case (US9600993) has mentioned in the specification (Col. 19, Line. 1-52) that through Fourier transforms and Laplace transforms (Fourier transforms , Laplace transforms) to perform shape or contour processing on human images for privacy protection. In addition, it is also mentioned in the instructions (Col. 20, Line. 11-34) to remove unwanted backgrounds (such as table lamps ), while retaining the desired The general shape or outline of the human figure, and, as mentioned in the instructions (Col. 22, Line. 1-26), changes in the person's position movement can be used to detect falling or sitting down events. For example, if the position of a person's head drops rapidly in a short period of time until it reaches the bottom edge of the field of view, it is judged that a person may fall.

However, in the conventional technology, if it is necessary to analyze the person's image in real time and perform Fourier transformation or Laplace transformation, it will cause a burden on the system's calculation and processing. If there are many people to be cared for, the system will be difficult to Burdening multiple people with real-time image processing will cause the system to crash. On the other hand, when the unwanted background is removed, and when the person being cared for stumbles over a chair, since the system has already removed the background, it will be difficult to determine the cause of the person being cared for. Causes a fall, and then, if only the position movement change is used as the basis for judging a fall, it is easy to cause a misjudgment.

Therefore, how to design a smart nursing care method that can protect the safety and privacy of the care recipient without requiring the care recipient to wear any detection device, and also increase the accuracy of judging the fall of the care recipient, all of which are This is the issue and focus of this case.

One object of the present invention is to provide a fall and posture recognition method with safe care and high privacy processing that allows the care recipient to be protected in safety and privacy.

Another object of the present invention is to provide a fall and posture recognition method with safe care and high privacy processing that increases the accuracy of judgment of falls of the person being cared for.

The present invention provides a fall and posture recognition method with safe care and high privacy processing, which is used in a fall, posture, and behavior recognition system, and provides identification and differentiation of the physical state of at least one person being cared for in at least one area. The fall and posture recognition system includes an image capture unit, an image processing unit and a fall recognition unit. The image capture unit is configured corresponding to the area and is used for shooting toward the person being cared for. The image processing unit is connected with the signal to the image capture unit, and the fall recognition unit is connected with the image processing unit with a signal, wherein the fall and posture recognition method includes the following steps: a) the image capture unit Capturing body images of the care recipients to generate body image data corresponding to each of the care recipients; b) the image processing unit receives the body image data, and performs image quality and noise preprocessing on the body image data; and c) the fall identification unit receives the image preprocessed Body image data and changes in human body posture are used to determine whether the person being cared for has fallen. If the person being cared for has fallen or has other abnormal conditions, an immediate notification message will be issued.

As mentioned above, the fall and posture recognition method, wherein the image processing unit includes an illumination change analysis module and an ambient light source description module, wherein the step b) includes the following steps: b1) the illumination change analysis module Detect changes in the ambient light source of the area to generate a pair of real-time information describing the ambient light source of the field; b2) The ambient light source description module promptly corrects and establishes a surrounding ambient light source model based on the real-time information of the light source; and b3) The image processing unit searches and detects and identifies multiple care recipients existing in the environment in the area according to the surrounding environment light source model situation.

The above fall and posture recognition method, wherein the fall recognition unit includes a care recipient classification module and a care recipient analysis module, wherein the step c) includes the following steps: c1) the care recipient The person classification module identifies the care recipients in the environment of the area based on the image processing unit's search detection and differentiation of the area, and establishes a correspondence corresponding to the identified care recipients in each of the environments. Skeleton points and a target rectangular tracking frame; c2) The care recipient classification module determines whether the identified care recipients in the environment match a characteristic of the care recipient, i.e., retrieves the identification information, and if so, establishes a a corresponding candidate box of possible objects; and c3) The care receiver analysis module retrieves and identifies information based on the characteristics, and analyzes and identifies the posture, gait and/or behavior of each care receiver.

As mentioned above, the fall and posture recognition method, wherein the care recipient classification module further includes a spatio-temporal filter sub-module, wherein the step c1) includes the spatio-temporal filter sub-module of step c11) The module provides appropriate logic and mechanisms, mask values and feature thresholds to calculate the specific features of the identified care recipients in the environment.

The above fall and posture recognition method, wherein the care recipient analysis module includes a geometric analysis sub-module, a curvature algorithm sub-module, a shape algorithm sub-module, and a human body local configuration algorithm sub-module Group (Body Part Algorithm), wherein the step c3) includes a step c31), the geometric analysis sub-module analyzes and identifies the shape, size and/or size change of the identified care recipient in the environment, and the curvature algorithm The sub-module identifies and calculates the posture changes of the specific curvature of each of the care recipients, the shape algorithm sub-module calculates and compares the specific body posture of each of the care recipients, and the human body local configuration algorithm sub-module The system calculates the various human body parts of each person being cared for. On the other hand, it can also establish a multi-person tracking algorithm model at the same time to facilitate simultaneous fall and posture recognition when there are multiple people in the scene.

As mentioned above, the fall and posture recognition method, wherein the care recipient analysis module further includes an optical flow algorithm sub-module, a bag of words algorithm sub-module (Bag of Word), a human skeleton and joint point sub-module, A machine learning sub-module, a fall detection algorithm sub-module, and a scale-invariant feature transformation algorithm module, wherein step c3) includes the optical flow algorithm sub-module of step c32) after step c31) The module performs motion detection, object cutting, calculation of collision time and object expansion, motion compensation coding and/or row stereometric measurement of the images of each of the care recipients. The bag-of-words algorithm sub-module is a comprehensive analysis of the fall process. Multiple feature vector groups corresponding to the posture are encoded and classified. The human body skeleton and joint point sub-module retrieves the curvature algorithm sub-module, the shape algorithm sub-module and the human body local configuration algorithm sub-module. Each calculation result generates a state detailed information corresponding to all activities and postures of each care recipient. The machine learning sub-module uses a neural network to perform deep learning to train, adjust and optimize the fall. Identification unit, the fall detection algorithm sub-module retrieves the curvature calculation Each calculation result of the method sub-module, the shape algorithm sub-module, the human body local configuration algorithm sub-module, the optical flow algorithm sub-module and the bag-of-words algorithm sub-module establishes a fall detection Together with the judgment model, the scale-invariant feature transformation algorithm module performs image compensation for distance changes of each of the care recipients in the area. In addition, this case can integrate and use various AI algorithms and data sets (Data Sets) training methods to improve the recognition rate and reduce the chance of false misjudgment.

In addition to the fall and posture recognition method mentioned above, the present invention relates to a fall and posture recognition system with safe care and high privacy processing. It also includes an intelligent visual electronic fence function that can prevent other accidents of the person being cared for. detection report. Intelligent visual electronic fence module, wherein the step b) includes a step b1'), the intelligent visual electronic fence module is each of the images captured by the image capture unit (IP CAM) 1 Establish a virtual electronic enclosure area in the area, and perform image superposition with the body image data of the person under care, wherein step c) includes a step c1') to determine whether the person under care has entered the area. If the care recipient enters within or leaves the electronic containment zone, or breaks into a restricted area restricted areas, etc., it will be determined that the person under care may enter the restricted dangerous area, stay for too long, or abnormally leave the safe area set by the electronic fence day and night, etc., and if confirmed, the real-time notification will be released immediately. Notify the message.

The fall and posture recognition method as mentioned above, wherein the image processing unit includes a visual privacy leakage prevention algorithm module, wherein the step b) includes a visual privacy leakage prevention algorithm module of step b1″) The group performs feature contour processing and dynamic image algorithm masking processing on the body image data of the care recipient to generate a human body synthetic mask that corresponds to the care recipient and has secure privacy masking.

As mentioned above, the fall and posture recognition method, wherein the step b1") includes a step b11") of performing feature contour processing and dynamic image algorithm masking processing on multiple object images in the area to generate multiple corresponding objects. A composite mask of the object described above with a secure privacy mask.

As mentioned above, the fall and posture recognition method, wherein the image processing unit further includes a night-time fall and posture model, wherein the step a) includes a step a1") and the night-time fall and posture model through fuzzy technology (Fuzzy Theory) analyzes and intelligently identifies and enhances the body image data at night.

As mentioned above, the fall and posture recognition method further includes a communication unit, and the communication unit is signal-connected to the fall recognition unit. The fall and posture recognition method further includes step d). The communication unit is used to send a caring inquiry message to each of the persons being cared for. It can also understand the on-site situation of the persons being cared for in an emergency and conduct two-way calls.

As mentioned above, the fall and posture recognition method, wherein the care recipient analysis module includes an abnormality detection unit for people in the room, an abnormality detection unit for a specific state for a long time, a work and rest statistics unit and a history browsing unit , wherein the step c3) includes the following steps: c31') the abnormality detection unit of someone in the house determines whether the person under care is in the predetermined area at a first predetermined time; c32' ) The abnormality detection unit in which the specific state is too long determines whether the person under care has remained motionless outside of a second predetermined time, or whether it has been in the area for too long; c33') The above-mentioned work and rest statistics unit continuously and automatically records the home life and work and rest status of each of the described care recipients; and the history browsing unit described in c34') is for browsing the home life and work and rest status of each of the described care recipients.

This case is a fall and posture recognition method with safe care and high privacy processing. It uses an image capture unit to provide intelligent image care for each care recipient. It can improve the accuracy of the fall judgment of the care recipient and enable the care recipient to While security and privacy are protected, medical labor costs can also be reduced, and all the above purposes can be achieved.

1:Image capture unit

2:Image processing unit

21: Illumination change analysis module

22:Ambient light source description module

23: Intelligent visual electronic fence module

24: Anti-visual privacy leakage algorithm module

3: Fall identification unit

4: Communication unit

31:Care recipient classification module

311: Spatiotemporal filter sub-module

32:Care recipient analysis module

321:Geometric analysis submodule

322: Curvature algorithm submodule

323: Shape algorithm submodule

324: Human body local configuration algorithm submodule

325: Optical flow algorithm sub-module

326: Bag of words algorithm submodule

327:Human skeleton and joint idea module

328:Machine learning sub-module

329: Fall detection algorithm module

320: Scale-invariant feature transformation algorithm module

331: Anomaly detection unit for people in the house

332: Anomaly detection unit in a specific state for too long

333: Work and rest statistics unit

334: Historical browsing unit

S101~S104, S1021~S1023, S1031~S1034, S10341, S10342, S101’, S1011”, S1012”: steps

Figure 1 is a block diagram of a fall and posture recognition system with safety care and high privacy processing according to the first embodiment of this case; Figure 2 is a fall and posture recognition system with safety care and high privacy processing in Figure 1 A flow chart of the posture recognition method; Figure 3 is a block diagram of a fall and posture recognition system with safety care and high privacy processing according to the second embodiment of this case; Figure 4 is a fall and posture recognition system with safety care and high privacy processing in Figure 3 and a flow chart of a fall and posture recognition method with high privacy processing; Figure 5 is a block diagram of the care recipient analysis module of the fall and posture recognition system in Figure 3; Figures 6 and 7 are the fall and posture recognition system in Figure 3 The schematic diagram of the curvature change of the curvature algorithm sub-module; Figure 8 is the shape description schematic diagram of the shape algorithm sub-module of the fall and posture recognition system in Figure 3; Figure 9 is the caregiver analysis model of the fall and posture recognition system in Figure 3 An extended block diagram of the group; Figure 10 is an extended flow chart of a fall and posture recognition method with safety care and high privacy processing in Figure 4; Figure 11 is a shape algorithm sub-module of the fall and posture recognition system in Figure 3 The distribution curve of the bag-of-words algorithm submodule; Figure 12 is a flow diagram of the machine learning sub-module of the fall and posture recognition system in Figure 3; Figure 13 is a fall and posture recognition system with a fall and posture recognition method with safety care and high privacy processing according to the third embodiment of the present case The block diagram; Figure 14 is a flow chart of a fall and posture recognition method with safety care and high privacy processing in Figure 13; Figure 15 is the intelligent visual electronic fence module deployment of the fall and posture recognition system in Figure 13 A schematic diagram of the electronic isolation block; Figure 16 is a block diagram of a fall and posture recognition method and posture recognition system with safety care and high privacy processing according to the fourth embodiment of this case; Figure 17 is a block diagram of a fall and posture recognition system with safety care and high privacy processing in Figure 16 A flow chart of a fall and posture recognition method with high privacy processing; Figure 18 is a block diagram of a fall and posture recognition method and posture recognition system with safety care and high privacy processing according to the fifth embodiment of this case; Figure 19 is a block diagram of Figure 18 A partial flow chart of a fall and posture recognition method with safe care and high privacy processing; Figure 20 is a block diagram of the care recipient analysis module of the fall and posture recognition system of Figure 17; Figure 21 is a continuation of Figure 20 The subsequent flow chart of the caregiver analysis module; and Figures 22 and 23 are schematic diagrams of conventional fall detection.

A fall and posture recognition method with safety care and high privacy processing according to the first embodiment of the present invention, as shown in FIG. 1 , is used in a fall and posture recognition system and is used to identify and distinguish at least one area. The physical state of at least one caregiver (for example, the elderly), in which the fall and posture recognition system includes an image capture unit (IP CAM) 1, an image processing unit (pre-processing) 2 and a fall recognition unit ( detected human) 3, the image capture unit (IP CAM) 1 is configured corresponding to the above-mentioned area and is used to capture images toward the person being cared for, and the image processing unit (pre-processing) 2 is in conjunction with the image capture unit (IP CAM) )1 signal connection, the fall recognition unit (detected human) 3 is connected with the image processing unit (pre-processing) 2 signal. The image capture unit 1 in this case uses a visible light camera that is commonly used by the general public and is low-cost, but uses image destructive processing technology to achieve a high privacy design.

Referring to FIG. 2 , initially in step S101 , the image capture unit 1 captures body images of the care recipients to generate body image data corresponding to each care recipient. Then in step S102 , the image processing unit 2 Receive body image data, and perform image quality and noise preprocessing on the body image data. Then in step S103, the fall identification unit 3 receives the body image data after image preprocessing, and determines that the person is being cared for through changes in human body posture. Whether the person fell down, if not, then return to step S101 and continue to collect images of each person being cared for. If it is determined that the person has fallen, then in step S104, an immediate notification message is issued and the relevant caregivers are notified to come to the scene immediately for treatment. Emergency assistance.

According to a second embodiment of the present invention, a fall and posture recognition method with safety care and high privacy processing is shown in FIG. 3. In this example, the image processing unit 2 includes an illumination change analysis module (illumination change estimation). ) 21 and an ambient light source tuning model 22, as shown in Figure 4 together. In this example, step S102 is followed by step S1021, where the illumination change analysis module 21 detects the environment of the area. The light source changes to generate a pair of real-time information describing the field environment light source, thereby adjusting and compensating the environment light source to provide a stable light source image with better quality. In step S1022, the environment light source description module 22 uses the real-time information of the light source. , timely correction and Establish a surrounding environment light source model to facilitate effective processing of the activity status images of the care recipients in the area collected by the image capture unit 1. In step 1023, the image processing unit 2 searches and detects the environment based on the surrounding environment light source model. Multiple identified care recipients.

In addition, the fall identification unit 3 may also include a care recipient classification module (Object Classification) 31 and a care recipient analysis module 32, as shown in Figure 5. In this example, the care recipient classification module 31 It further includes a spatio-temporal filter sub-module (Spatio-Temporal Filter) 311, and the care recipient analysis module 32 includes a geometric analysis sub-module (Geometric Analysis) 321, a curvature algorithm sub-module (Curvature Algorithm) 322, and a shape Algorithm sub-module (Shape-based Algorithm) 323 and a human body partial configuration algorithm sub-module (Body Part Algorithm) 324.

Step S1023 is followed by step S1031. Based on the image processing unit's search and detection of the area, the care recipient classification module 31 identifies and distinguishes the identified care recipients in each environment of the area, and establishes corresponding identifications in the environment. The corresponding skeleton point of the person under care and a target rectangular tracking frame. There is a step S10311 after step S1031. The spatio-temporal filter sub-module (Spatio-Temporal Filter) 311 provides appropriate logic and mechanism, mask value (Mask) and features. The threshold value (Threshold) is used to calculate the specific characteristics of the environment to identify the care recipient, so as to filter certain specific objects or filter interference or noise to the object to be detected, and then step S1032, classifying the care recipient The module 31 determines whether the identified care recipients in the environment match one of the characteristics of the care recipients, that is, the identification information. If not, return to step S1031 to continue searching and detection. If they match, proceed to step S1033. , establish a corresponding candidate box (Bounding Box) of possible objects, and in step S1034, the care recipient analysis module 32 analyzes and identifies the posture, gait and/or behavior of each care recipient based on the feature extraction, that is, the recognition information. By combining the above modules with other algorithms, it is possible to avoid the situation where people overlap but are misjudged as falling. There is a step S10341 after step S1034, in which the geometric analysis sub-module 321 analyzes and identifies the environment. The shape, size and/or size changes of the identified care recipients are identified and calculated by the curvature algorithm sub-module (Curvature Algorithm) 322 to assist in identifying and calculating the specific curvature posture changes of each care recipient. Tasks such as finding the target of the care recipient, posture recognition, gait recognition, and behavior recognition are calculated and compared with the specific body posture of each care recipient by the shape-based algorithm sub-module (Shape-based Algorithm) 323, and are calculated by the local configuration of the human body Method module (Body Part Algorithm) 324 calculates each body part of each care recipient.

Assume that when a human body falls backward or forward, the body trunk should have specific curvature changes when specific posture changes (as shown in Figures 6 and 7). Therefore, the curvature algorithm sub-module 322 can change the human body posture through , and changes in body movements, etc., to help understand whether the elderly or cared for who are currently concerned may have a tendency to fall, and the changes in the curvature of the human body are undoubtedly important information, and through the human body Continuous changes in curvature can help understand the real-time changes in the human body's body shape and skeleton, and can help to instantly grasp the necessary specific process conditions and specific real-time changes that may meet or satisfy a fall when the human body is walking or moving in the environment. The main function of the shape algorithm sub-module 323 is to collect corresponding "shape descriptions" that keep in mind the different behaviors of the human body (as shown in Figure 8) and complement and complement the aforementioned curvature algorithm sub-module 322), and together provide a method for identifying different postures of the human body. It is part of the detection standards and appearance basics to provide feature recognition algorithms to assist in detecting specific behavioral appearance changes (such as walking, falling, etc.) when human body posture changes.

Referring also to Figure 9, in this example, the care recipient analysis module 32 further includes an optical flow algorithm sub-module (Optical Flow Algorithm) 325, a bag of word algorithm sub-module (Bag of Word) 326, Human skeleton and joint point Algorihm (Bone & articulation point Algorihm) 327, a machine learning sub-module (Artificial Intelligent & machine learning) 328, a fall detection algorithm sub-module (Fall Detection AI Algorithm) 329, a scale-varying Scale-invariant feature transform algorithm module (Scale-invariant feature transform, SIFT) 320.

In addition, as shown in Figure 10, there is a step S10342 after step S10341, in which the optical flow algorithm sub-module (Optical Flow Algorithm) 325 performs motion detection, object cutting, collision time and object expansion on the image of the person being cared for. The calculation, motion compensation encoding and/or row stereometric are performed by the bag-of-words algorithm sub-module 326. The posture of the fall process Corresponding multiple feature vector group codes are established and classified. The human skeleton and joint point submodule 327 retrieves the calculation results of the curvature algorithm submodule 322, the shape algorithm submodule 323, and the human body local configuration algorithm submodule 324. , generating a state detailed information corresponding to all activities and postures of each care recipient, and the machine learning sub-module 328 performs deep learning using a neural network to train, adjust and optimize the judgment of the fall identification unit 3. The measurement algorithm sub-module 329 captures the calculations of the curvature algorithm sub-module 322, the shape algorithm sub-module 323, the human body local configuration algorithm sub-module 324, the optical flow algorithm sub-module 325 and the bag-of-words algorithm sub-module 326 As a result, a fall detection and judgment model is established. The scale-invariant feature transformation algorithm module 320 performs image compensation for changes in the distance of each care recipient in the area. On the other hand, a multi-person tracking algorithm model can also be established at the same time. , to facilitate simultaneous fall and posture recognition when there are multiple people in the scene.

Among them, the optical flow algorithm sub-module 325 performs calculations through optical flow (Optical flow or optic flow). Optical flow refers to the speed at which an image of an object expresses movement. The reason why an object can be detected by the human eye when it is moving is because when the object moves, a series of continuously changing images will be formed on the human retina. These changing information will continue to flow through the retina of the eye at different times. , just like a kind of light flowing through, so it is called optical flow. The optical flow method can be used to describe the movement of human bodies or objects in the environment relative to machine vision (equivalent to the observer), which can be regarded as the movement of the surface or edge of the elderly or the person being cared for (the object being observed). In other words, optical flow is caused by the movement of the foreground object itself in the scene, the movement of the camera, or the mutual movement of the two. To do this, you can first assign a velocity vector (optical flow, including size and direction) to each pixel (Pixels) of each image in the environment, so that an optical flow field can be formed. If there are no moving objects in the image in the environment, the optical flow field will be continuous and uniform. However, if there are moving objects, the optical flow of the moving objects and the optical flow of the image will be different, and the optical flow field will no longer be continuous and uniform, thus It can detect moving objects and their positions. The optical flow method is very useful in graphic recognition, computer vision and other image processing fields. It can be used for motion detection, calculation of collision time and object expansion, or for specific three-dimensional measurements through the surface and edges of objects. , in the continuous process of falling, the human body will simultaneously produce changes in the limbs, skeleton, and joints. By detecting and identifying changes in the optical flow field, information on human body posture changes can be obtained at the same time, and combined with the aforementioned algorithms to assist in fall recognition and associated posture identification and analysis.

The bag-of-words algorithm submodule 326 is a probabilistic model of Bayesian multi-layer architecture. It can provide a classification method that needs to establish an intelligent learning mechanism for various objects in the field. It can be used to help determine and evaluate the possibility of falling. and auxiliary mechanisms, and then collect these various clips (Video Clip) related to the fall, and map these various clips related to the fall to their corresponding codebooks (Codebook). Therefore, in the elderly or cared for in their fields and activities, the distribution curve of these code words (Codeword) that may appear can be obtained and statistically calculated. For example, the physical meaning shown in Figure 11 can represent the occurrence of specific gestures. Distribution curve of times. Curve graphs are often used to analyze image content characteristics in image processing and computer vision. From the concept of a curve graph, we can use it to analyze the continuous changes in a specific posture and the judgment factors or principles that may lead to falls.

The human skeleton and joint point sub-module 327 captures the aforementioned curvature algorithm sub-module 322, human body local configuration algorithm sub-module 324, shape algorithm sub-module 323, optical flow algorithm sub-module 325 and bag-of-words algorithm sub-module Relevant data of module 326 is used to obtain all activities and posture details of the elderly or cared for in the environment, and algorithms are designed and integrated with these detailed information to facilitate the construction of the human body's skeleton and joints in the environment. Points and their key parameters. When all real-time important parameters such as human skeletons and joint points in these environments are generated, it is equivalent to understanding the relationship between various objects and human bodies in the scene at any time, and the existence of specific human bodies and their personalized activity postures. Details, as well as the real-time interactive relationship between the human body and the human body. Based on the real-time parameters of these skeletons and joint points, the activities and process information of the people of concern in the environment can be judged based on the real-time information grasped, so as to facilitate the implementation of specific behaviors and actions. Judgment, such as standing, sitting, getting up, lying down, squatting, falling,... various actions. However, because the specific behavior of the human body is derived from a series of postures, it is actually very complex. In other words, different specific behaviors (Behavior) may often have the same specific posture (Pose). For example, the specific "throw forward" action (shown in Figures 5 and 6), Some continuous gestures are actually very similar to "bending over", "hunchbacking", "leaning forward while holding something", etc. We can call it "pseudo-fall" here. It is easier for human beings, who are the spirits of all things, to quickly distinguish the subtle differences in these movements because they have intelligence. But if you want computers or machines to be able to distinguish such problems at the same time, you must use artificial intelligence methods such as machine learning and deep learning to design these algorithms and build machines. The learning rules and models enable the machine to have artificial intelligence designated with specific capabilities.

Referring to Figure 12, which is a schematic flow chart of the machine learning sub-module 328, you must first select a framework (or platform) for designing this machine learning algorithm. Commonly used frameworks such as Tensorflow, Poytch, etc. or fast feature embedding volumes Product structure CNN, etc. For example, in the case of Tensorflow, this framework can be regarded as a machine learning design tool, which contains many available algorithms and functions; in the case of Tensorflow, it mainly provides some methods of deep learning model (Deep Learning model).

The fall detection algorithm sub-module 329 captures the aforementioned curvature algorithm sub-module 322, shape algorithm sub-module 323, human body local configuration algorithm sub-module 324, optical flow algorithm sub-module 325 and bag-of-words algorithm. Relevant data of module 326 is used to obtain detailed information on all the activities and postures of the elders or people being cared for in the environment, and algorithms are designed to facilitate the integration of these detailed information to construct the skeleton, joints and joints of the human body in the environment. Its key parameters, when all the real-time important parameters of the human body skeleton and joint points in these environments are generated, it is equivalent to knowing at any time the relationship between various objects and the human body in the scene, the existence of a specific human body and its personalized activity posture details, and The interactive relationship between the human body and the human body. In addition, the fall detection algorithm sub-module 329 module has a built-in AI model established by deep learning (Deep Learning) and other methods. Its input parameters are combined with information from the aforementioned modules (such as human skeleton and joint point modules). etc.), in addition to capturing the information output by the aforementioned modules, it also considers the changes in parameters such as the current position, speed, and acceleration of each joint point of the human body, and uses the established AI model to conduct a series of fall detection. Test and judge.

Scale-invariant feature transformation algorithm module 320 needs to perform scale space extrema Detection, key point positioning, direction determination and key point description. The scale space extreme value detection is to search for image positions on all scales. Potential interest points that are invariant to scale and rotation are identified through the Gaussian differential function; key point positioning is to determine the position and scale at each candidate position through a well-fitted model. Key points are selected based on their degree of stability; direction determination is based on the local gradient direction of the image, assigning one or more directions to each key point position. All subsequent operations on the image data are transformed relative to the direction, scale and position of the key points, thereby providing invariance to these transformations; the key point description is in the neighborhood around each key point, in the selected Measure the local gradient of the image on a scale. These gradients are transformed into a representation that allows relatively large local shape deformations and illumination changes.

According to another third embodiment of the present invention, a method for combining fall and posture recognition with safe care and high privacy processing with an intelligent visual electronic fence module is shown in Figures 13 and 14. In the embodiment of the intelligent visual electronic fence, the image capture unit captures the image of the person being cared for in the environment. After passing through the image processing unit, the highly private streaming image is removed. In addition to being sent to the fall identification unit for fall identification, another identical stream image will be sent to the smart visual electronic fence module to determine whether the smart visual electronic fence module is triggered to prevent various possible obstructions. Dangerous situation. In the intelligent visual electronic fence module (AI Electrical Fence Algorithm) 23, as in step S101 before step S101, the intelligent visual electronic fence module 23 is established in each area captured by the image capture unit (IP CAM) 1 An electronic isolation block 230 is superimposed with the body image data of the person being cared for; refer to Figure 15 as well, and determine whether the person being cared for 4 is within the range of the electronic isolation block 230 in step S103' or Stay away from the electronic isolation block 230, determine whether the person under care has abnormal behavior, and if so, issue an immediate notification message.

Among them, the intelligent visual electronic fencing module 23 is an integration of AI smart algorithm and security electronic fencing technology to construct an "AI electronic fencing algorithm" to expand its application in smart homes, smart long-term care, or Used for safe care of human-centered areas. This case can integrate the range setting of electronic fences based on special concerns and application needs. In addition to assisting in fall determination and other multi-identification applications, for example, the range of electronic fences can be set to be illuminated. If the person being cared for is within the range of the electronic fence, a danger is likely to occur, so report it immediately. Therefore, computer vision and image processing technology can be used to establish an interactive interference relationship between the target of concern (the person being cared for) and the "visual electronic fence". When the interactive interference relationship is identified and established, the system will immediately issue a notification .

According to another fourth embodiment of the present invention, a fall and gesture recognition method with safety care and high privacy processing is shown in FIG. 16 and FIG. 17 together, in which the image processing unit 2 includes an algorithm module to prevent visual privacy leakage. Group (Privacy Protection Algorithm) 24, and as in step S1011 after step S101, the anti-visual privacy leakage algorithm module 24 performs feature contour processing and dynamic image algorithm masking processing on the body image data of the care recipient to generate A pair of destructive masks of human body images corresponding to the care recipient and with secure privacy masking. Furthermore, after step S1021", the feature contour processing and dynamic image algorithm masking processing of multiple object images in the area can be performed as in step S1012". To generate a plurality of object image destructive masks corresponding to the object and having a secure privacy mask.

Among them, the anti-visual privacy leakage algorithm module 24 provides the integrated use of "image processing" and "AI algorithm" technology to perform feature contour masking processing on images from visible light cameras, as well as dynamic image algorithm superposition, and finally synthesize the image. Achieve high recognition rate while taking into account the safety and privacy of people-centered care. On the other hand, each care recipient target in the environment of concern can be similarly masked to avoid leaking the detailed characteristics and outline of individual objects. , to achieve further privacy protection.

According to another fifth embodiment of the present invention, a fall and posture recognition method with safety care and high privacy processing is shown in Figures 18 and 19, in which the image processing unit 2 further includes a night fall and posture model 24, and in After step S101, as in step S1011, the nighttime fall and posture model 24 enhances the body image data through fuzzy theory analysis and intelligent identification.

The fall and posture recognition system further includes a communication unit 4 (such as mobile App/Line), which is connected to the fall recognition unit 3, and after step S104, as in step S1041, The caregiver sends a care inquiry message to each person being cared for through the communication unit 4. The caregiver can also learn about the on-site situation of the person being cared for from the App or Line in an emergency and conduct a two-way conversation.

Referring to Figures 20 and 21 together, in this example, the care recipient analysis module 32 includes an abnormality detection unit 331 for people in the house, an abnormality detection unit 332 for a specific state that has been too long, a work and rest statistics unit 333, and A history browsing unit 334 may also include the following steps after step S1023. S1301 The abnormality detection unit 331 of someone in the house determines whether the person under care is in a predetermined area at a first predetermined time. If someone enters the home of the person under care, Space, the patent of this invention further integrates the related technologies such as human body posture recognition and electronic fencing disclosed above to automatically display on the system when someone enters the home space state of the person being cared for, and push the prompt message to the caregiver. on the reader’s mobile App and Line.

As in step S1302, the abnormality detection unit 332 of the prolonged specific state determines whether the person being cared for has remained motionless outside a second predetermined time (such as sleeping time), or whether the person has been in the area for too long, in order to further pay attention to the person being cared for. The caregiver is active in various situations in the living space, and can ensure that the person being cared for is in a stable and good state of activity. The patent of this invention further uses the aforementioned human body posture recognition related technology to understand whether the person being cared for is in a specific state or whether the fixed movements are correct. There is anomaly detection that lasts too long.

In step S1303, the work and rest statistics unit 333 continues to automatically record the home and life work and rest status of each person being cared for. In order to further pay attention to the home and life work and rest status of the care recipient, the patent of the present invention further integrates the human body posture recognition and other related information disclosed above. Technology that automatically and continuously records the home life and rest status of the person being cared for (activity status and analysis of postures such as standing/sitting/lying, etc.), and displays it on the system to facilitate caregivers to understand and track and observe at any time, and to facilitate evaluation and measurement of the person being cared for. The caregiver's daily healthy physiological status.

Finally, in step S1304, the history browsing unit 334 is used to browse the home life and rest status of each person being cared for. In order to simultaneously track and pay attention to the long-term work and rest status of the person being cared for, the patent of the present invention further integrates the related technologies such as human body posture recognition disclosed above. , and automatically and long-term recording of the daily routine of the person being cared for (for example, it can be stored and can record the 30-day historical activity browse automatically). This case is a fall and posture recognition method with safe care and high privacy processing. It uses an image capture unit to perform smart image care on each care recipient, and increases the accuracy of the care recipient's fall judgment through smart learning. It can also use intelligent The recognition of the visual electronic fence module can set the range of the electronic fence to a place where the care recipient rarely goes (such as an idle space where debris is stored). If the care recipient is within the range of the electronic fence, it is easy to If danger may occur, report it immediately. On the other hand, it can also prevent visual privacy leakage algorithm modules, perform privacy shielding on care recipients and environmental objects, and reduce medical labor costs, achieving all the above purposes.

The present invention shows that it is completely different from the conventional technology in terms of purpose, means and efficacy, and is a major breakthrough. However, it should be noted that the above embodiments are only illustrative to illustrate the principles and effects of the present invention, and are not intended to limit the scope of the present invention. Anyone skilled in the art can make modifications and changes to the embodiments without violating the technical principles and spirit of the present invention. The protection scope of the present invention shall be as described in the patent application scope described below.

S101~S104: Steps

Claims (10)

A fall and posture recognition method with safe care and high privacy processing, which is used in a fall and posture recognition system and is used to identify and distinguish the physical state of at least one care recipient in at least one area. The fall and posture recognition system includes An image capture unit, an image processing unit and a fall recognition unit. The image capture unit is an Internet camera (IP CAM). The image capture unit is configured corresponding to the areas and is provided for facing the persons under care. The image processing unit is connected with the image capturing unit through signals, and the fall recognition unit is connected through signals with the image processing unit. The fall and posture recognition method includes the following steps: a) The image capturing unit captures Obtain the body images of the care recipients to generate a body image data corresponding to each care recipient; b) The image processing unit receives the body image data and performs image quality and noise prediction on the body image data. Processing; and c) The fall identification unit receives the body image data after image preprocessing and determines whether the care recipients have fallen through body posture changes. If the care recipients fall or have other abnormal conditions, Publish an instant notification message; wherein the image processing unit includes an illumination change analysis module and an environmental light source description module, wherein step b) includes the following steps: b1) the illumination change analysis module detects the environment of the area The light source changes to generate a pair of real-time information describing the field environment light source; b2) The environmental light source description module promptly corrects and establishes a surrounding environment light source model based on the real-time information of the light source; and b3) The image processing unit searches and detects and identifies multiple care recipients existing in the environment based on the surrounding environment light source model; wherein, the fall identification unit includes a care recipient classification module and a care recipient The patient analysis module, wherein step c) includes the following steps: c1) The caregiver classification module identifies the caregiver in the environments in the region based on the image processing unit's search, detection and distinction in the region , and establish corresponding skeleton points and a target rectangular tracking frame corresponding to each identified care recipient in the environment; c2) The care recipient classification module determines whether any of the identified care recipients in the environment matches the characteristics of the care recipients. A feature is retrieved to identify information, and if it matches, a corresponding candidate box of possible objects is established; c3) The care recipient analysis module analyzes and identifies the gestures of each of the care recipients based on the feature retrieved and identified information. , gait and/or behavior; wherein the care recipient analysis module includes a curvature algorithm sub-module, wherein the step c3) includes a step c31), the curvature algorithm sub-module identifies and calculates each of the care recipients Attitude changes of a specific curvature. For the fall and posture recognition method described in item 1 of the patent application, the caregiver classification module further includes a spatio-temporal filtering sub-module, wherein the step c1) includes a step c11) and the spatio-temporal filtering sub-module provides Appropriate logic and mechanisms, mask values and feature thresholds to calculate the specific features of the identified care recipients in those environments. The fall and posture recognition method described in item 1 of the patent application, wherein the care recipient analysis module includes a geometric analysis sub-module, a shape algorithm sub-module and a human body local configuration algorithm sub-module, wherein In step c31) the geometric analysis sub-module analyzes and identifies The shape, size and/or size change of the cared persons identified in the environments, the shape algorithm sub-module calculates and compares the specific body posture of each care recipient, the human body local configuration algorithm sub-module The group calculates each part of the human body of the person being cared for. For the fall and posture recognition method described in item 5 of the patent application, the care recipient analysis module further includes an optical flow algorithm sub-module, a bag-of-word algorithm sub-module, and a human skeleton and joint point sub-module. , a machine learning sub-module, a fall detection algorithm sub-module, and a scale-invariant feature transformation algorithm module, wherein step c3) includes a step c32) of the optical flow algorithm sub-module after step c31) The group performs motion detection, collision time and object expansion calculation on each image of the person being cared for. The bag-of-words algorithm sub-module establishes multiple feature vectors corresponding to the posture during the fall process, codes and classifies the human skeleton and joint points. The module retrieves each calculation result of the curvature algorithm sub-module, the shape algorithm sub-module and the human body local configuration algorithm sub-module, and generates a state detailed information corresponding to all activities and postures of the care recipient, The machine learning sub-module uses a neural network to perform deep learning to train, adjust and optimize the judgment of the fall identification unit. The fall detection algorithm sub-module captures the curvature algorithm sub-module, the shape algorithm sub-module module, the human body local configuration algorithm sub-module, the optical flow algorithm sub-module and the bag-of-words algorithm sub-module to establish a fall detection and judgment model, the scale-invariant feature transformation algorithm The module performs image compensation for changes in the distance of each care recipient in the area. The fall and posture recognition method described in item 1 of the patent application, wherein the image processing unit includes an intelligent visual electronic fence module, and the fall and posture recognition method includes a step a before step a) '), the intelligent visual electronic fencing module creates an electronic fencing block in each area captured by the image capture unit, wherein step c) includes an Step c1') Determine whether the care recipients are within the range of the electronic containment block or away from the electronic containment block. If the care recipients enter the range of the electronic containment block or leave the electronic containment block, Encircle the block, determine if there are abnormal phenomena or behaviors among those under care, and issue the real-time notification message. The fall and posture recognition method as described in item 1 of the patent application, wherein the image processing unit includes an algorithm module for preventing visual privacy leakage, wherein the step b) includes a step b1") of the anti-visual privacy leakage The algorithm module performs feature contour masking processing and dynamic image algorithm masking processing on the body image data of the care recipient to generate a pair of destructive masks of the body image of the care recipient with secure privacy masking. For the fall and posture recognition method described in item 6 of the patent application, the step b1") includes a step b11") that performs feature contour processing and dynamic image algorithm masking processing on multiple object images in the area to generate Multiple object image destructive masks corresponding to those objects with secure privacy masks. The fall and posture recognition method described in item 1 of the patent application, wherein the image processing unit further includes a night-time fall and posture model, wherein step a) includes a step a1") the night-time fall and posture model uses fuzzy technology Analysis and intelligent identification enhance the body image data. The fall and posture recognition method described in item 1 of the patent application, wherein the fall and posture recognition system further includes a communication unit, the communication unit is signal-connected to the fall recognition unit, and the fall and posture recognition method further includes a Step d) The communication sheet YuanGong sends a caring inquiry message to each person being cared for, and can also understand the on-site situation of the person being cared for in an emergency and conduct two-way calls. For the fall and posture recognition method described in item 1 of the patent application, the care recipient analysis module includes an abnormality detection unit for people in the house, an abnormality detection unit for a specific state for a long time, a work and rest statistics unit, and A history browsing unit, wherein the step c3) includes the following steps: c31') the abnormality detection unit of the person in the house determines whether the care recipients are in the predetermined area at a first predetermined time; c32') the specific The abnormality detection unit in an excessively long state determines whether the persons being cared for have remained motionless outside of a second predetermined time, or whether they have been in the area for too long; c33') The work and rest statistics unit continues to automatically record each The home life and rest status of the person being cared for; and c34') The history browsing unit is for browsing the home life and rest status of each person being cared for.