TWM625774U - Care-giving detection system based on deep image-based behavior analysis of care recipients - Google Patents

Abstract

A care detection system based on depth image-based behavior analysis of the care recipient, the care detection system includes: a depth photography care detection device and a care background management system. The depth photographic care detection device is used to be installed at the care recipient's residence to obtain depth image data in the care recipient's residence, and to perform behavior analysis processing of the care recipient on the depth image data to determine the care recipient's behavior state, And the state information representing the behavior state of the care recipient is output through wired or wireless means. The care background management system is connected in communication with the deep photography care detection device, and is used for generating notification information according to the status information and sending it to at least one terminal device of the caregiver of the care recipient.

Description

Nursing detection system based on deep image-based behavior analysis of care recipients

This work is about the image-based care detection system, more particularly, a care-based detection system based on deep image-based behavior analysis of the care recipients.

Currently on the market for care devices, wearable sensors or pressure sensors are commonly used for fall detection or bed exit detection.

Wearable products are popular because of their low cost. However, the care recipients (such as patients or elders) often do not like to wear them due to their dignity or discomfort, and the effect is not good. The sensors are often bumped or squeezed. Sensitivity may be compromised by pressure sensors; light weight, uncooperative or immobile patients may affect sensitivity. In addition, smart mattresses have high prices and safety considerations in wiring.

To monitor the situation in the ward, it is necessary to install 2D image surveillance video, but 2D image surveillance makes people have no privacy protection. Therefore, the current care device still needs to be improved.

One of the objectives of the present invention is to provide a care detection system based on depth image-based behavior analysis of the care recipient, which utilizes a depth photographic care detection device to be installed in the care recipient's residence to obtain the depth image data in the care recipient's residence, and Behavior analysis processing of the care recipient is performed on the deep image data to judge the behavior state of the care recipient. By this technology, the real-time status of the care recipient can be grasped and notified to the caregiver accordingly, so as to help the caregiver to give the care recipient in a timely manner. Caregivers help, while taking into account the privacy and dignity of the care recipients.

In order to achieve at least the above purpose, a care detection system based on depth image-based behavior analysis of the care recipient is proposed. The care detection system includes: a depth photography care detection device and a care background management system. The depth photographic care detection device is used to be installed in a care recipient's residence to obtain depth image data in the care recipient's residence, and to perform a care recipient behavior analysis process on the depth image data to determine the care recipient Behavior status, and output the status information representing the behavior status of the care recipient through wired or wireless means. The care background management system is connected in communication with the deep photography care detection device, and is used for generating notification information according to the status information and sending it to at least one terminal device of the caregiver of the care recipient.

In an embodiment of the present invention, the depth photography care detection device includes: a depth camera unit, a communication unit, a storage unit, and a data processing unit. The depth camera unit is used for generating the depth image data. The communication unit is used for communication and connection with the nursing background management system. The storage unit is used for storing data. The data processing unit is electrically coupled with the depth camera unit, the communication unit, and the storage unit, and is used for performing the behavior analysis processing of the care recipient on the depth image data to determine the behavior state of the care recipient, and The state information representing the behavior state is sent to the nursing background management system through the communication unit.

In an embodiment of the present invention, the data processing unit is configured to perform the behavior analysis processing of the care recipient on the depth image data to determine the behavior state of the care recipient, and the behavior state includes at least one of a plurality of states, These states include the state of getting out of bed, the state of falling, and whether or not to go out.

In an embodiment of the present invention, the depth photographic care detection device further includes: a depth image conversion module, a scene floor plane detection module, a bed state estimation module, and a human body state position detection module . The depth image conversion module is used for coordinate conversion of the depth image data. The scene floor plane detection module is used for determining the scene floor plane in the depth image data according to the floor plane setting. The bed state estimation module is used for estimating the bed activity state of the care recipient based on the depth image data. The human body state position detection module is used for detecting the human body state and position based on the depth image data. The depth image conversion module, the scene floor plane detection module, the bed state estimation module, and the human body state position detection module are executed by the data processing unit to operate in the depth photography care detection device.

In an embodiment of the present invention, the depth photography care detection device is further configured to output a depth image stream to the care background management system, so as to send the information to the caregiver's caregiver through the care background management system. at least one terminal device.

In an embodiment of the present invention, the care background management system is further used to provide a device management interface for setting the entry and exit position or attention area of the care recipient's residence. Perform the behavior analysis processing of the care recipient on the depth image data at the entry and exit position or attention area within the department to judge the behavior state of the care recipient.

In an embodiment of the present invention, the care background management system is further used to record and analyze the behavior status of the care recipient and send it to the at least one terminal device of the care recipient's caregiver.

In some embodiments of the present invention, the nursing background management system includes: a network unit, a memory unit and a control unit. The network unit is used for communication and connection with the depth photography care detection device. The memory unit is used to store data. The control unit is electrically coupled to the network unit and the memory unit, and is used for receiving the status information output by the depth photography care detection device, generating the notification information according to the status information, and sending the notification information through the network unit to The notification information is sent to the at least one terminal device of the caregiver of the care recipient.

In some embodiments of the present invention, the control unit is further configured to receive a depth image stream output by the depth photography care detection device, and send the depth image stream to the care recipient through the network unit according to the depth image stream the at least one terminal device of the user.

In some embodiments of the present invention, the depth photographic care detection device is arranged on a wall of the care recipient's residence to obtain depth image data in the care recipient's residence.

The above embodiments provide a care detection system based on depth image-based behavior analysis of the care recipient. The depth photographic care detection device is used for being installed in the care recipient's residence to obtain the depth image data in the care recipient's residence, and performing the care recipient behavior analysis processing on the depth image data to judge the care recipient's behavior state. The care background management system is used for generating notification information according to the status information and sending it to at least one terminal device of the caregiver's caregiver. With this technology, the real-time status of the care recipient can be grasped and notified to the caregiver accordingly, so that the caregiver can help the careee in a timely manner, and at the same time, the privacy and dignity of the careee can be taken into account.

In order to fully understand the purpose, features and effects of this creation, hereby use the following specific embodiments and cooperate with the attached drawings to give a detailed description of this creation, and the description is as follows:

Please refer to FIG. 1 , which is a schematic diagram of an embodiment of a care detection system 10 according to the present invention. As shown in FIG. 1 , the care detection system 10 is used to analyze the behavior of the care recipient based on depth images. The care detection system 10 includes a depth photography care detection device 100 and a care background management system 200 . The depth photographic care detection device 100 is configured to be installed at the care recipient's residence to obtain depth image data in the care recipient's residence, and perform behavior analysis processing of the care recipient on the depth image data to determine the care recipient's behavior status , and output the status information representing the behavior status of the care recipient through wired or wireless means. The care background management system 200 is connected in communication with the deep photography care detection device 100, and is used for generating notification information according to the status information and sending it to at least one terminal device 300 of the care recipient's caregiver.

In a practical application scenario, the care detection system 10 can be configured to include a plurality of depth photographic care detection devices 100, which are respectively installed in the residences of a plurality of care recipients to individually perform the behavior analysis processing of the care recipients to determine the plurality of care recipients. The behavior state of the care recipient, and individually outputting the state information representing the behavior state of the care recipient to the care background management system 200 through wired or wireless means. In addition, the care background management system 200 can also be configured to associate different care recipients with the terminal devices 300 of the corresponding caregivers, so as to generate notification information according to the status information and send it to the care recipient's caregiver's terminal device 300. In some embodiments, the care detection system 10 may be implemented in any medical unit such as a large hospital, a general hospital, a medical center, a clinic, a care center, etc. However, the present invention is not limited by these examples. For example, the care detection system 10 may also be suitable for use by individual households. In addition, in some embodiments, the care detection system 10 may be configured to be applied to different care recipients, for example, the care detection system 10 may be configured to be suitable for the elderly, persons requiring long-term care, patients or children, etc. , the implementation of this creation is not limited by these examples.

Thereby, the care detection system based on the depth image-based behavior analysis of the care recipient disclosed in FIG. 1 and FIG. 2 uses a depth photography care detection device to be installed at the care recipient's residence to obtain the depth image data in the care recipient's residence. , and perform the behavior analysis processing of the cared person on the deep image data to judge the behavior status of the cared person. By this technology, the real-time status of the cared person can be grasped and notified to the caregiver accordingly, so as to help the caregiver to timely Giving help to the person being cared for can also take into account the privacy and dignity of the person being cared for. In addition, the depth photography care detection device has both hardware and computing functions for acquiring depth image data and judging the behavioral state of the person being cared for. Management, especially in medical units such as large hospitals, general hospitals, medical centers, clinics, care centers, etc.

Various embodiments for implementing the care detection system 10 are provided below.

FIG. 2 is a schematic block diagram of an embodiment of the depth photographic care detection apparatus 100 of FIG. 1 . As shown in FIG. 2 , the depth photography care detection device 100 includes: a depth camera unit 110 , a communication unit 120 , a storage unit 130 and a data processing unit 140 .

The depth camera unit 110 , such as a 3D depth camera, is used to generate the depth image data; the depth image data may be a 3D depth image, such as a depth image stream or a plurality of depth images over a period of time.

The communication unit 120 is used for communication and connection with the nursing background management system 200; the communication unit is configured as a wired or wireless communication circuit, for example, supports Bluetooth or Bluetooth Low Energy, ZigBee, or a wireless local area network Wireless communication protocols such as wireless communication protocols, such as wireless communication protocols, are used to wirelessly communicate with other electronic devices, such as through wireless network devices (such as base stations, hubs, or router, etc.) to transmit the data to the nursing background management system 200, or communicate with the nursing background management system 200 directly.

The storage unit 130, such as flash memory, random access memory (RAM) or any suitable memory device, is used to store data, such as data generated by the depth camera unit or data in the process of judgment analysis and processing .

The data processing unit 140, such as a microprocessor and a microcontroller, is electrically coupled to the depth camera unit 110, the communication unit 120, and the storage unit 130, and is used for executing the care recipient on the depth image data The behavior analysis process is used to judge the behavior state of the care recipient, and send the state information representing the behavior state to the care background management system 200 through the communication unit 120 .

In one embodiment, the data processing unit 140 is configured to perform the behavior analysis process of the care recipient on the depth image data to determine the behavior state of the care recipient, and the behavior state includes at least one of a plurality of states. Including the state of being out of bed, the state of falling, and whether or not to go out. In another embodiment, the statuses may further include statuses as to whether the care recipient has been in the toilet for too long.

FIG. 3 is a schematic block diagram of another embodiment of the depth photographic care detection apparatus 100 of FIG. 1 . In an embodiment of the present invention, the depth photography care detection device 100 further includes: a depth image conversion module 410, a scene floor plane detection module 420, a bed state estimation module 430, and a human body state position Detection module 440. The depth image conversion module 410 , the scene floor plane detection module 420 , the bed state estimation module 430 , and the human body state position detection module 440 are executed by the data processing unit 140 to operate in the depth photography Inside the care detection device 100 .

The depth image conversion module 410 is used for coordinate conversion of the depth image data. For example, through the depth image conversion module 410, the imaged point cloud (Point Cloud Stitching) can be used to visualize scene information, and at the same time, the external parameter correction of the depth camera unit is completed, and coordinate conversion is performed to calculate each position in the scene space. For example, when the depth camera unit 110 is installed on the wall, the obtained depth image is at an acute angle with the floor plane; if the input depth image required by other modules such as the bed state estimation module 430 is from the ceiling When the depth image obtained from the angle is obtained, the depth image conversion module 410 can be configured to perform coordinate conversion on the depth image obtained by the depth camera unit 110 to obtain the depth image of the ceiling angle, so as to be output to the bed state estimation module 430 for further processing. judge.

The scene floor plane detection module 420 is used for determining the scene floor plane in the depth image data according to the floor plane setting. For example, through the scene floor plane detection module 420, the depth camera unit can automatically detect the floor plane of the scene, and set a 3D region of interest (ROI) according to the floor plane, so that the installer can mark the area in the line of sight. For sensitive areas, increase the flexibility of event judgment rules to improve judgment accuracy.

The bed state estimation module 430 is used for estimating the bed activity state of the care recipient based on the depth image data. For example, the bed state estimation module 430 is implemented using deep learning technology. The neural network-like model of supervised learning is used as the estimation engine. In the learning stage, the training data set is input, the pure depth information is used as the input data, and the corresponding character activity status in the bed range is the output data, so as to obtain the trained estimation data. test engine. The bed state estimation module 430 can be configured as a hardware or software module of the estimation engine that implements the training, and is used for judging the activity state of characters in the bed range, that is, to input pure depth information to the bed state estimation module 430 And judge the activity state of the characters in the bed range as the behavior state of the person being cared for.

The human body state and position detection module 440 is used to detect the human body state and position based on the depth image data. For example, through the human body state position detection module 440, the human body state and its position can be detected, and at the same time, whether a fall event has occurred can be judged in real time. For example, when the human body is outside the bed and the depth information of the head position indicates that the head is within a certain height range (eg, within 30, 40 or 50 cm) close to the floor plane, a fall event can be judged.

In addition, in some embodiments, as shown in FIG. 3 , the depth image conversion module 410 , the scene floor plane detection module 420 , the bed state estimation module 430 , and the human body state position detection module 440 It can also cooperate with an integrated module 450 to integrate the operations of the above modules. In addition, in some embodiments, the communication module 460 can be operated with the protocol, so as to realize the communication between the depth photography care detection device 100 and the care background management system 200, such as Message Queuing Telemetry Transport (Message Queuing Telemetry Transport, MQTT). However, the present creation is not limited to this.

FIG. 4 is a schematic diagram of an embodiment of an application scenario of the depth photographic care detection apparatus 100 in FIG. 1 . In some embodiments, the depth photographic care detection device 100 is used to be installed on a wall of the care recipient's residence to obtain depth image data in the care recipient's residence. As shown in FIG. 4 , when the depth photographic care detection device 100 is installed on the wall of the care recipient's residence, the time and cost of installation and maintenance can be greatly saved. Correspondingly, the depth photography nursing detection device 100 can be configured to perform the coordinate transformation of the depth image according to the setting manner of the depth photography nursing detection device 100 , as described in the above examples.

FIG. 5 is a schematic diagram of an embodiment of a depth image obtained by the depth photographic care detection device 100 of FIG. 1 . Since this creation uses 3D depth image technology, although the image only has the depth information of the object's position in space, the image can be used to detect the body shape of the human body or the shape of the object; as described in the previous embodiments, the visual image-based Sensing technologies, such as body posture analysis, fall status judgment, and activity status detection, use deep learning to achieve more accurate detection, so that the real-time status of patients can be grasped and the care recipients can be facilitated to receive timely help. Patient privacy and dignity.

In one embodiment, the depth photographic care detection device 100 is further configured to output a depth image stream to the care background management system 200, so as to send the information to the caregiver's caregiver through the care background management system 200 accordingly. At least one terminal device 300 . For example, the deep photography care detection device 100 and the care background management system 200 may be configured to use Real Time Streaming Protocol (RTSP) to perform control stream communication. However, the present creation is not limited to this example.

In some embodiments, the care background management system 200 may be implemented using one or more computing devices, computer systems or servers.

FIG. 6 is a schematic block diagram of an embodiment of the care background management system 200 of FIG. 1 . As shown in FIG. 6 , in some embodiments, the nursing background management system 200 includes: a network unit 210 , a memory unit 220 and a control unit 230 .

The network unit 210 is used for communication and connection with the depth photography care detection device 100; the network unit 210 is configured as a wired or wireless communication circuit, for example, supports Bluetooth or Bluetooth Low Energy, ZigBee, or wireless local area network and other wireless communication protocols to wirelessly communicate with other electronic devices, such as through wireless network devices (such as bases) located in or near the care recipient's residence (such as a medical unit, care unit, or home). station, hub, or router, etc.) to communicate with the depth photography care detection device 100, or the care background management system 200 directly communicates with the depth photography care detection device 100.

The memory unit 220, such as at least one of flash memory, random access memory (RAM), or any suitable memory or storage device, is used to store data, such as received from a depth camera care detection device 100 data, transfer data, record or compile the above data, or the care background management system 200 judges the data in the process of analysis and processing.

The control unit 230, such as a microprocessor and a microcontroller, is electrically coupled to the network unit 210 and the memory unit 220, and is used for receiving the status information output by the depth photography care detection device 100, and according to the status information Generate notification information, and send the notification information to the at least one terminal device 300 of the caregiver's caregiver through the network unit 210 .

In one embodiment, the care background management system 200 is further used to record and analyze the behavior status of the care recipient and send it to at least one terminal device 300 of the care recipient's caregiver. FIG. 7 is a schematic diagram of an embodiment of an interface provided by the care background management system 200 of FIG. 1 . 7 shows the behaviors of the care recipient at different times of a day recorded by the care background management system 200 according to the state information of the deep photographic care detection device 100, such as walking, resting in bed, indoors or outdoors, and in the bathroom.

In an embodiment of the present invention, the care background management system 200 is further configured to provide a device management interface for setting the entry and exit positions or attention areas of the care recipient. Behavior analysis processing of the care recipient is performed on the depth image data at the entry and exit position or attention area in the residence to judge the behavior state of the care recipient. Please refer to FIG. 8 , which is a schematic diagram of an embodiment of another interface provided by the care background management system 200 of FIG. 1 . For example, the caregiver can connect to the care background management system 200 through the terminal device 300, and then display the device management interface from the display screen of the terminal device 300 (such as a mobile phone or a computing device such as a computer). Personnel can mark sensitive areas in the line of sight, such as the entry and exit positions or attention areas in the careee's residence (as represented by circles in Figure 8), to increase the careee's behavioral state or event judgment rules (such as getting out of bed) The height of the body position after leaving the bottom indicates the possibility of falling) elasticity, thereby improving the accuracy of judgment.

In some embodiments of the present invention, the control unit 230 is further configured to receive the depth image stream output by the depth photography care detection device 100, and send the depth image stream to the person under care through the network unit 210 according to the depth image stream. the at least one terminal device 300 of the caregiver of the person.

As described above, the embodiments provide various implementations of a care detection system based on depth image-based care recipient behavior analysis. The depth photographic care detection device is used for being installed in the care recipient's residence to obtain the depth image data in the care recipient's residence, and performing the care recipient behavior analysis processing on the depth image data to judge the care recipient's behavior state. The care background management system is used for generating notification information according to the status information and sending it to at least one terminal device of the caregiver's caregiver. With this technology, the real-time status of the care recipient can be grasped and notified to the caregiver accordingly, so that the caregiver can help the careee in a timely manner, and at the same time, the privacy and dignity of the careee can be taken into account.

The present creation has been disclosed above with preferred embodiments, but those skilled in the art should understand that the embodiments are only used to describe the present creation, and should not be construed as limiting the scope of the present creation. It should be noted that all changes and substitutions equivalent to this embodiment should be set to be included in the scope of the present creation. Therefore, the scope of protection of this creation shall be determined by the scope of the patent application.

10: Care Detection System 100: Depth photography care detection device 110: Depth Camera Unit 120: Communication unit 130: storage unit 140: Data processing unit 200: Care background management system 210: Network Unit 220: Memory Unit 230: Control Unit 300: Terminal device 410: Deep Image Conversion Module 420: Scene floor plane detection module 430: Bed state estimation module 440: Human body state position detection module 450: Integrate Mods 460: Communication Module

FIG. 1 is a schematic diagram of an embodiment of a care detection system according to the present invention. FIG. 2 is a schematic block diagram of an embodiment of the depth photographic care detection device of FIG. 1 . FIG. 3 is a schematic block diagram of another embodiment of the depth photographic care detection device of FIG. 1 . FIG. 4 is a schematic diagram of an embodiment of an application scenario of the depth photographic care detection device of FIG. 1 . FIG. 5 is a schematic diagram of an embodiment of a depth image obtained by the depth photographic care detection device of FIG. 1 . FIG. 6 is a schematic block diagram of an embodiment of the care background management system of FIG. 1 . FIG. 7 is a schematic diagram of an embodiment of an interface provided by the care background management system of FIG. 1 . FIG. 8 is a schematic diagram of an embodiment of another interface provided by the care background management system of FIG. 1 .

10: Care Detection System

100: Depth photography care detection device

200: Care background management system

300: Terminal device

Claims (10)

A care detection system based on deep image-based behavior analysis of care recipients, the care detection system includes: A depth photographic care detection device is installed in a care recipient's residence to obtain depth image data in the care recipient's residence, and performs the care recipient behavior analysis process on the depth image data to determine the care recipient's behavior Behavioral state, and output the state information representing the behavioral state of the care recipient through wired or wireless means; and A care background management system is connected in communication with the deep photography care detection device, and is used for generating notification information according to the status information and sending it to at least one terminal device of a caregiver of the care recipient. The care detection system according to claim 1, wherein the depth photography care detection device comprises: a depth camera unit for generating the depth image data; a communication unit for communicating with the care background management system; a storage unit for storing data; and a data processing unit, electrically coupled to the depth camera unit, the communication unit, and the storage unit, for performing the behavior analysis processing of the care recipient on the depth image data to determine the behavior state of the care recipient, and through The communication unit sends status information representing the behavior status to the nursing background management system. The care detection system according to claim 2, wherein the data processing unit is configured to perform the behavior analysis processing of the care recipient on the deep image data to determine the behavior state of the care recipient, and the behavior state includes at least one of a plurality of states. One, the states include the state of getting out of bed, the state of falling, and the state of going out or not. The care detection system according to claim 2 or 3, wherein the depth photography care detection device further comprises: a depth image conversion module for performing coordinate conversion on the depth image data; a scene floor plane detection module for determining the scene floor plane in the depth image data according to the floor plane setting; a bed state estimation module for estimating the bed activity state of the care recipient based on the depth image data; and a human body state position detection module for detecting the human body state and position based on the depth image data; The depth image conversion module, the scene floor plane detection module, the bed state estimation module, and the human body state position detection module are executed by the data processing unit to operate in the depth photography care detection device . The care detection system according to claim 2 or 3, wherein the depth photography care detection device is further configured to output a depth image stream to the care background management system, so as to send it to the care recipient through the care background management system the at least one terminal device of the caregiver. The care detection system according to claim 1, wherein the care background management system is further configured to provide a device management interface for setting the entry and exit position or attention area of the care recipient's residence, and the depth photography care detection device is based on the The in-and-out position or attention area in the caregiver's residence performs the behavior analysis processing of the careee on the depth image data to judge the behavior state of the careee. The care detection system according to claim 6, wherein the care background management system is further used to record and analyze the behavior status of the care recipient and send it to the at least one terminal device of the care recipient's caregiver. The care detection system according to any one of claim 1, 6, and 7, wherein the care background management system includes: a network unit for communicating with the depth photographic care detection device; a memory unit for storing data; and a control unit, electrically coupled to the network unit and the memory unit, for receiving the status information output by the depth photography care detection device, generating the notification information according to the status information, and passing the network unit to the status information The notification information is sent to the at least one terminal device of the caregiver of the care recipient. The care detection system according to claim 8, wherein the control unit is further configured to receive a depth image stream output by the depth photography care detection device, and send the depth image stream to the care recipient through the network unit according to the depth image stream the at least one terminal device of the caregiver. The care detection system as claimed in claim 1, wherein the depth photographic care detection device is arranged on a wall of the care recipient's residence to obtain depth image data in the care recipient's residence.

Images