AU2020104377A4 - Intelligent Safe Home System for the Elderly People - Google Patents

Abstract

Intelligent Safe Home System for the Elderly People As elderly people need care and support to lead a safe life without worries and anxiety, it is very important today with regard to older people. Lack of knowledge of evolving behavioral trends of elderly people at home contributes to their relatives' harassment of them. We have established a viable home protection device for the elderly to be deployed in their homes here. We developed a smart home protection system that includes pedestrian monitoring, face recognition and fall detection using open-source hardware to help with cameras and networks. We use the KNN model context subtraction method on the basis of the open source OpenCV library to detect moving objects and combine hog-svm to create a pedestrian tracking module. To extract facial characteristics, the trained vggnet-16 neural network model is used, and then a face recognition module is designed, which can be used for international alarm intrusion. The original caffer model was updated to mobilenet model for human motion recognition based on the original openpose. Information on the location of 6 key points on the body trunk was obtained at 18 key points, and the role of fall detection was achieved by combining the SVM classifier. By integrating the GSM module, the old man's home and fall details would be a first time input to the old man's family members, who can completely guarantee the old man's safety. The fall behavior recognition performance of face recognition is strong, the face recognition rate can reach 85 percent, the fall behavior recognition rate can reach more than 90 percent, and the fall false alarm rate is less than 10 percent for strangers and elders, according to our experiment. The proposed method should, therefore, be implemented in real life. Str Video Capturu'ng Is HumianN detected? Y Is that humnan a stranger? Y N Send the Alert to the Family N Does the elder fall downY Fig. 2

Description

Str

Video Capturu'ng

Is HumianN detected?

Y

Is that humnan a stranger? Y

N Send the Alert to the Family

N Does the elder fall downY

Fig. 2

Editorial Note 2020104377 There is 8 pages of Description only.

TITLE OF THE INVENTION Intelligent Safe Home System for the Elderly People.

FIELD OF THE INVENTION

[001]. The present disclosure is generally related to an Intelligent Safe Home System for Protecting the Elderly People from foreign interference, accidental falling down, illness and other abnormal circumstances.

BACKGROUND OF THE INVENTION

[002]. The primary objective of this invention is to develop an intelligent, secure home system for the elderly, based on openCV. This invention includes five main modules such as real-time monitoring module, tracking module for human body recognition, face recognition module, behavior recognition module, warning module, etc. The real-time image stream is collected by the monitoring module in real time, and the image is processed on the basis of openCV. Three modules such as human body recognition, face recognition and behavior recognition for recognition activity are transmitted to the processed image, and the living conditions of the elderly at home are evaluated according to the target feature information. The device is constructed to collect abnormal information, such as foreign interference, falling down of old people, illness of old people and other abnormal circumstances, the abnormal information would be transmitted by network contact to the elderly family. Through this invention, the family member of the older people can access real-time information about the elderly at any time.

SUMMARY OF THE INVENTION

[003]. As elderly people need care and support to lead a safe life without worries and anxiety, it is very important today with regard to older people. Lack of knowledge of evolving behavioral trends of elderly people at home contributes to their relatives' harassment of them. We have established a viable home protection device for the elderly to be deployed in their homes. We developed a smart home protection system that includes pedestrian monitoring, face recognition and fall detection using open-source hardware to help with cameras and networks.

[004]. The entire system's operating theory is shown in the fig. 1. Video controls the acquisition of real-time video data and transmits the data to the computer for processing, and then the computer transmits the processed images to the measurement and recognition modules for relevant face recognition, pedestrian detection and activity detection. Products of identification and analysis are returned to the machine. The machine analyzes the results and, if there is any risk, sends dangerous information to the appropriate cell phones.

[005]. The device consists mainly of five modules: a preprocessing image acquisition module, a pedestrian monitoring module, a face recognition module, a gesture recognition module and an alarm module.We use the KNN model context subtraction method on the basis of the open source OpenCV library to detect moving objects and combine hog-svm to create a pedestrian tracking module.

[006]. The machine regulates the GSM module to give the family cell phone dangerous information after discrimination. If assessed as the elderly, the image information for behavior judgment will be passed to the behavior recognition module. The recognition module currently only has the purpose of determining the dangerous actions of dropping. If the old man is judged to fall, the risky details would be sent by GSM to the old man's relatives. The device was tested and found that up to 85% accuracy was achieved.

DETAILED DESCRIPTION OF THE INVENTION

[0071. The aged, however, still have a number of injuries at home. When old age grows, old people's bodies can grow weaker and weaker. The old people also have little resistance when facing the invasion and decline of outsiders. They might not even call out for help or ask for help while facing danger. They can also skip the best chance of rescue and ultimately lose their lives because they are saved by no one. We therefore need a smart home security system that can monitor elderly activities at home in a timely manner and provide immediate assistance. Applying the smart home device to the home is possible to simplify and make the monitoring task more intelligent. Also, when strangers enter or the old collapse, the device can timely track the elderly at home and timely feedback information, which is more reliable. Moreoverr, the scheme is more affordable and safe as opposed to bringing the elderly to a nursing home and asking for help.

[0081. Nowadays, with the Internet of Things' growing popularity, the smart home device is not remote. A small number of devices and open source programs can create a smart home system at home that is usable. We developed a smart home security system for the elderly based on open source Raspberry Pi hardware, supported by some cameras and wireless networks. The neural network is constructed by Tensor Flow to process images on the basis of the open source OpenCV library, and the face recognition module and the body posture recognition module are created. Then perform the following functions urgently necessary by the elderly, the intrusion alarm of strangers and the fall alarm of the elderly.

[009]. Combined with the APP cell phone, the details about the home and the collapse of the elderly can be fed back to the elderly family for the first time, which completely guarantees the safety of the system's elderly and then I can finally provide the potential system design enhancement.

[0010]. Proposed System Design: The entire system's operating theory is shown in the fig. 1. Video controls the acquisition of real-time video data and transmits the data to the computer for processing, and then the computer transmits the processed images to the measurement and recognition modules for relevant face recognition, pedestrian detection and activity detection. Products of identification and analysis are returned to the machine. The machine analyzes the results and, if there is any risk, sends dangerous information to the appropriate cell phones. The device consists mainly offive modules: a preprocessing image acquisition module, a pedestrian monitoring module, a face recognition module, a gesture recognition module and an alarm module.

[0011]. Image acquisition and preprocessing: The first step is to use the monitor in the key areas of the home to gather real-time video information. In the second step, because the collected video information is large, the calculation amount will be too large to ensure good real-time performance if all the data is processed, so the key frames of the collected video information will be extracted. Process the frame image of the key. In this innovation, the key frame image acquisition frequency is 15 images per second. Third, since various modules have distinct image processing specifications, we perform distinct preprocessing of key frame images. We copy the main frame into two images here.

[0012]. Some of them retain a higher resolution, and to increase the image contrast, pass them through the sharpening filter. It is used primarily in the face recognition module, since to effectively recognize face details, this module requires a higher resolution. The other image can be scaled and its resolution can be properly reduced, which can effectively reduce the human tracking and behavior recognition module computation and at the same time maintain its recognition capacity.

[0013]. Fourth, the image will be processed according to the software flow, and the processed frame image will be transferred for processing to various modules. Preprocessing, mainly through sharp filtering and subtraction of the context. The sharpening filter can be used to enhance the accuracy of the image. In this invention, the Laplacian operator is primarily used to explain the edges, contour lines and face area image information and increase the rate of face recognition.

[0014]. All moving body tracking and identification are required by the human body tracking module. Background subtraction is a kind of technique that is commonly used in current technologies for detecting motion targets, and can effectively track moving objects and say whether a space has a moving object. So, to remove the foreground image, this innovation uses background subtraction. Mainstream names include adaptive mixed Gaussian model, KNN model, etc. Context subtraction For extracting moving objects from pictures, the KNN model is selected.

[0015]. Pedestrian detection and tracking module: The device uses a directional gradient histogram-support vector machine (HOG-SVM) human body recognition algorithm to detect and identify human targets efficiently. HOG and SVM are blended into the algorithm. By observing the contrast between the contour and the context, the primary concept is to perform human body recognition and there can be different appearances of different individuals wearing different clothing, but the contours of the human body are identical. The location of the human body can be accurately defined and monitored based on this HOG-SVM. This innovation is based on OpenCV for human body recognition using the trained HOG-SVM model in the openCV library, which greatly reduces the complexity of growth.

[0016]. Face recognition module: The face recognition module consists mainly of four parts: acquisition and identification of face images, preprocessing of face images, extraction of face image features, matching and recognition.

[0017]. Acquisition and identification of face images: Image acquisition can pick ordinary cameras, mount multiple cameras at home in a fixed position to cover all areas, take pictures to collect images, and use the pedestrian detection and tracking module to identify and capture human body position. In the preceding step, we need to cut the face from the preprocessed image. So we need to train the face detector first, and then use the face detector to detect the face in the image.

[0018]. We are using the Viola-Jones Face Detector here. This approach is based upon the form of sliding window. In order to decide if the area of the window contains human faces, a fixed size window is used to slide the range into our image. If the picture has several faces, pick the window with the highest likelihood of cutting out the face. Finally, the image of the acquired face is transmitted to the next processing stage. Preprocessing of face images: The image acquired is preprocessed, such as enhancing the image, de drying, sharpening, and so on.

[0019]. The mean filtering method is used in this invention to remove noise from the image. Mean filtering can effectively remove image noise, weaken the effect of light on image, weaken image sharp points and retain precise characteristics. Extraction of facial image features: To extract face features, Convolutional Neural Networks (CNN) is used. While conventional methods of image processing can to some degree extract facial features, far from the level of realistic implementation, the accuracy of extracting facial features in face recognition is still not reached. Including the key points of face detail, such as eye, ear, nose, mouth and even eyebrow characteristics and position, face characteristics extracted by the CNN method can well reflect the face.

[0020]. The comprehension rate of human beings has also been surpassed by some research focused on in-depth learning. In this invention, the neural network used is vggnet-16. The architecture of the vggnet neural network is a convolutional neural network proposed by Oxford University's computer vision department. Its performance is superior. Vggnet-based studies and implementations are numerous and are relatively mature. The VGGFace face database is used to train vggnet-16, and the face recognition accuracy rate is more than 95 percent . For feature extraction, the trained model is used. Matching and recognition: The machine takes images automatically when a person steps into the field of vision of the camera.

[0021]. The acquired image is extracted and preprocessed by image by face field. For face feature extraction, the preprocessed face is inserted into the qualified Vggnet-16 model. The face eigenvalues measured are balanced and compared to those in the database. Euclidean distance is determined in this innovation for matching recognition.

[0022]. Human pose Recognition Module: The primary role of the module for the identification of human motion is to recognise human behaviors and actions and to recognize and warn those dangerous behaviors and actions. They are frail and vulnerable to falling for older people, particularly when there is only one elderly person in the family. He was unable to contact the police after the old man collapsed. The main goal of this module is behavioral awareness of the fall of the elderly.

[0023]. The device control warning module sends relevant information to the cell phones of the elderly's relatives when the result is recognized as a fall, so that the elderly can be rescued quickly. In this innovation, openpose training model is adopted based on openCV to input human images into the openpose model, obtain the key position of the human body, and input the key position to the SVM classifier to evaluate the SVM classifier Proposed by Carnegie Mellon University's computer lab, Openpose can define key points of the human skeleton from two-dimensional images easily and accurately. Openpose will identify the location information of 18 key human body points and use the key human body information points together with the SVM classifier to assess if individuals are dropping.

[0024]. OpenPost Human Attitude Recognition Project is an open source library focused on convolutionary neural network and supervised learning and caffe platform developed by Carnegie Mellon University (CMU). The Openpose algorithm is a human bottom-up skeleton algorithm for posture. First, the F function of the original image is extracted by vggnet-19 in the main body network, and then two branches are used to regress the S position of the joint points and the L direction of the pixels in the skeleton. Multi-stage iteration is the resulting branch network structure.

[0025]. Every stage calculates a loss function and links the node's location S and the direction L of the pixels in the branch 1 skeleton and the original image features extracted for training in the next stage by vggnet-19

[0026]. For the whole stage, the process can be expressed as St = pt(F, St-1, Lt-1), M t 2and L' = O(FSt-1,Lt-1),Mt 2, here pt and t represent respectively the position S of the T stage and the convolutional neural network of the skeletal direction L.

Two loss functions are introduced in the process of regressing the position S of the joint points and the direction L of the pixels in the skeleton. The L2 loss function is used between the correct value and the predicted value as f, = W(p).S(p) - SJ| and fj = Z 1 ZW(p).IL (p) - L71, Sj* and L* separately represent the real labeled values of PCM and PAFs in the image. Because the position P in the original image is not marked, W(P) = 0.

[0027]. The overall objective function can be derived as ft = Ei(ft + f). The open pose recognition rate is very high in the process of use, but the amount of computation is very large, only under good hardware conditions to ensure a good real-time. The open pose, therefore, needs to be changed. The original opening is USES vgg-19, and the model of trained coffee is wide and around 200 M in size.

[0028]. Here, the size of the mobileNet model used by openCV is only 7.8MB, which, in the event of a small loss of recognition degree, substantially decreases the size of the model and the measurement number. Therefore, on the 2.8GHz Quad-core 7 CPU performance, mobile enet-thin achieves 4.2fps faster than vgg-19 model fps, which is thought to perform better on GPU and be more important to real life. In terms of accuracy of recognition, the recognition rate on the mobilenet training model is 81.3 percent, which is lower than 94.2 percent of vgg-19 due to computation reduction.

[0029]. In this invention, the concept is altered on the basis of an open source kit. The original author can change the model to run openpose on the CPU, which can run openpose on mobile devices, but its efficiency is worse. The high resolution of the measurement quantity is found to be large in the phase of usage, and the real-time output is low. The resolution is too poor, the real-time output is fine, but the rate of recognition decreases rapidly, the human location can not be calculated, simple to false alarm.

[0030]. It has excellent real-time performance and high reliability, with a resolution of 368x368. There are five different movements, including side stand, front stand, front open arms, side squat, side crouch, etc. Openpose can correctly define the location of the skeleton, whether it is front or side, squat or lie down. According to the relative modifications of the skeleton position, the fall recognition module will analyze whether or not the elderly fall.

[0031]. Openpose primarily measures 18 primary positions of the human body, while in order to assess the condition of the body, we only need 6 main trunk positions, such as nose, spine, left hip, right hip, left knee, right knee. Six point data can be trained by the classifier SVM, and it is possible to use the trained model to decide whether to fall. The SVM classifier is based on the statistical principle of small samples, which means that even small samples can produce good results, and the lack of training samples is currently an important problem in the field of fall detection, so using the SVM classifier can mitigate this problem to some degree.

[0032]. In addition, the SVM kernel function can handle high-dimensional data sets very well, so that the SVM can easily handle the high-dimensional joint features extracted by the previous openpose without decreasing the classifier's accuracy and generalization capability due to its high latitude. The potential state of the video is divided here into two categories: regular state and fallen state.

[0033]. The regular state is that elders usually walk and sit; the fall state is that elders lie on the ground. In this invention, we pick the kernel function as a Gauss kernel function when training SVM classifier, the kernel function parameter (gamma) is 0.05, and the penalty parameter is 1. The accuracy of the SVM classifier for joint state classification reached 93.25 percent after training.

[0034]. Alarm module: This innovation adopts the GPRS module as an alarm module, realizes serial communication control of GPRS and realizes the sending of short messages. The system transmits the relevant alarm information to the set mobile phone number through the GSM module to realize the alarm feature when the data generated by the above-mentioned face recognition and attitude recognition modules are assessed as hazardous by the system.

[0035]. Work flow of the system: The camera gathers the video data in real-time and transmits the data for processing to the computer. For operation recognition, the device transmits the processed images to the appropriate face recognition, pedestrian detection and behavior detection modules. The product of an interpretation of the identification is returned to the machine. The machine analyzes the outcomes. If danger occurs, by regulating the GSM module, it sends dangerous information to the appropriate cell phone. The flow chart is shown in Fig. 2.

[0036]. The camera gathers image information and processes openCV-based image information. To decide if there is an individual in the image information, the processed image information is transmitted to the pedestrian detection and tracking module. The pedestrian information image is passed into the face recognition module for discrimination if there is a male. It sends an alert to the machine if the face recognition module is judged to be a stranger.

[0037]. The machine regulates the GSM module to give the family cell phone dangerous information after discrimination. If assessed as the elderly, the image information for behavior judgment will be passed to the behavior recognition module. The recognition module currently only has the purpose of determining the dangerous actions of dropping. If the old man is judged to fall, the risky details would be sent by GSM to the old man's relatives. The device was tested and found that up to 85% accuracy was achieved.

[0038]. In this invention we successfully developed an intelligent, secure home system for the elderly, based on openCV. Five main modules are included in the entire system: real time monitoring module, tracking module for human body recognition, face recognition module, behavior recognition module, warning module, etc. The real-time image stream is collected by the monitoring module in real time, and the image is processed on the basis of openCV.

[0039]. Three modules of human body recognition, face recognition and behavior recognition for recognition activity are transmitted to the processed image, and the living conditions of the elderly at home are evaluated according to the target feature information. Once the device is constructed to collect abnormal information, such as foreign interference, falling down of old people, illness of old people and other abnormal circumstances, the abnormal information would be transmitted by network contact to the elderly family. At the same time, the GSM module can be upgraded and fitted with mobile applications, such as Android. The older members of the family can also use the client at any time to access real-time information about the elderly.

Editorial Note 2020104377 There is 1 page of Claims only.

Claims (6)

1. CLAIMS: We Claim: 1. We claim that invention Safe Home System for Protecting the Elderly People will be useful in protecting elders from foreign interference, accidental falling down, illness and other abnormal circumstances includes.
2. 2. As we claimed in 1, the primary objective of this invention is to develop an intelligent, secure home system for the elderly, based on openCV.
3. 3. We claim that the invention includes five main modules such as real-time monitoring module, tracking module for human body recognition, face recognition module, behavior recognition module, warning module, etc.
4. 4. As we claimed in 1 and 3, the real-time image stream is collected by the monitoring module in real time, and the image is processed on the basis of openCV.
5. 5. We claimed in 1, in this invention the device is constructed to collect abnormal information, such as foreign interference, falling down of old people, illness of old people and other abnormal circumstances and the abnormal information would be transmitted by network contact to the elderly family.
6. 6. Through this invention, the family member of the older people can access real-time information about the elderly at any time.

   Fig. 1

   Fig. 2