JP3238765U - Posture/Action Recognition System - Google Patents

Abstract

A posture/action identification system capable of monitoring actions such as getting out of bed, falling, and the like. A posture/action identification system includes at least one imaging device (1) and at least one background server (2). An imaging device captures three-dimensional depth image data within the target environment. A background server receives the three-dimensional depth image data captured by the imaging device. The background server implements a depth learning application, performs image detection and tracking based on the 3D depth image data, captures the human head position and the bed-type object position, and based on the human head position, in the target environment In addition to determining the number of people in the bed, based on the overlapping state of the position of the human head and the position of the bed-type object, the state on the bed is determined, and the non-on-bed state is determined based on the drop width and drop width time of the human head. . [Selection drawing] Fig. 1

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a posture and motion identification system, and more particularly to a posture identification system that determines on-bed and/or non-on-bed conditions and sends them to electronic devices to provide warnings and notifications.

For elderly or frail patients, falls are the leading cause of injury and death occurring in hospitals and nursing homes. Falls are more likely to occur when patients leave their beds, but given the ever-increasing demand for medical and nursing care, it was virtually impossible for nursing staff to pay constant attention to patients. .

On the other hand, if a patient leaves the hospital bed, the above-mentioned problems could be solved if bed separation conditions, falls or other actions could be monitored and immediate warnings and notifications could be provided. Therefore, based on the human head position and the bed-type object position, the number of people in the target environment, the state on the bed, and the state not on the bed can be determined, and the posture state can be transmitted to the electronic device for warning and notification. technology was required.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a posture and motion identification system that can monitor bed-leaning, falls, and other behaviors when a patient leaves the bed.
Another object of the present invention is to determine the number of people in the target environment, the state on the bed, the state not on the bed, and the state of movement of the bed position based on the position of the human head and the position of the bed-type object, and electronically record the posture state. To provide a posture/motion identification system capable of sending warnings and notifications to devices.

In order to solve the above problems, according to a first aspect of the present invention, there is provided a posture/motion identification system comprising at least one imaging device and at least one background server, wherein the imaging device captures 3D depth image data in a target environment, the background server receives the 3D depth image data captured by the imaging device, the background server is implemented with a depth learning application; performing image detection and tracking based on the three-dimensional depth image data, capturing a human head position and a bed-type object position, determining the number of people in a target environment based on the human head position, and A posture/movement characterized by judging a state on the bed based on the overlapping state of the body position and the position of the bed-type object, and judging a state not on the bed based on the fall width and the fall width time of the human head. An identification system is provided.

The depth learning application comprises a depth training unit, an image receiving unit, an image detection and tracking unit and a state determination unit, wherein the depth training unit performs model training on multiple field depth data stored in the background server. to build a depth learning model, the image receiving unit receives the three-dimensional depth image data, the image detection and tracking unit is connected with the depth training unit and the image reception unit, and performs image detection and tracking. The tracking unit includes a depth learning model, and performs image detection and tracking on the three-dimensional depth image data via the depth learning model to detect the human head position and the bed-shaped object position in the target environment. and tracking a human head and a bed-shaped object, wherein the state determination unit is connected with the image detection and tracking unit to determine the number of people in the target environment based on the human head position; It is preferable to determine the on-bed state based on the overlapping state of the head position and the bed-shaped object position, and to determine the non-on-bed state based on the fall width and the fall width time of the human head.

Preferably, the three-dimensional depth image data is composed of three color channels of RGB or composed of grayscale image channels, and the number of input bits of the three-dimensional depth image data is 8 bits or 16 bits.

The posture/action identification system of the present invention has the following effects (1) and (2).
(1) When the patient leaves the bed, it is possible to monitor the action of leaving the bed, falls and other actions.
(2) Based on the position of the human head and the position of the bed-type object, determine the number of people in the target environment, the state on the bed, the state not on the bed, and the movement state of the bed position, and send the posture state to the electronic device for warning. and notification.

1 is a block diagram showing the overall configuration of a posture/motion identification system according to an embodiment of the present invention; FIG. 1 is a block diagram showing the configuration of a background server of a posture/motion identification system according to an embodiment of the present invention; FIG. FIG. 2 is a block diagram showing the configuration of a depth learning application of the posture/motion recognition system according to an embodiment of the present invention; 1 is a flow diagram of a posture and motion recognition system according to an embodiment of the present invention; 1 is a flow diagram of a posture and motion recognition system according to an embodiment of the present invention; 1 is a flow diagram of a posture and motion recognition system according to an embodiment of the present invention;

Other technical contents, features and effects of the present invention will be clarified in the detailed description of the preferred embodiments with reference to the drawings.

Please refer to FIGS.
FIG. 1 is a block diagram showing the overall configuration of a posture/motion identification system according to an embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of the background server of the posture/motion identification system according to one embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of the depth learning application of the posture/motion recognition system according to one embodiment of the present invention.
As shown in FIGS. 1 to 3, the posture/motion identification system according to one embodiment of the present invention includes at least one imaging device 1 and at least one background server 2. As shown in FIGS. The imaging device 1 is a depth imaging device that captures three-dimensional depth image data in a target environment, or a device with depth imaging capabilities.

The 3D depth image data (eg, point cloud) represents a data aspect of the range information of the environment received by the depth camera. A depth image mainly represents the outline of an object, is more confidential than a color image, and has the advantage of being able to use the "depth information" that it has, so that it can be applied to a confidential space.

The 3D depth image data may consist of RGB tricolor channels or may consist of a grayscale image channel. A single-channel depth image (grayscale image) is generated by projection using a projection algorithm, the depth image represents the distance information of the object, and a general color image is formed by RGB three color channels, but the number of channels is too large. However, AI model detection takes a lot of time and consumes a lot of system resources. Therefore, if it can be modified to accept single channel, single channel can release unnecessary resource consumption, reduce time per algorithm and increase overall performance compared to 3 channels. can.

The 3D depth image data has an input bit number (each pixel of the depth image) of 8 bits or 16 bits, and represents the detected distance information from 0 to 65535. Since the data is input after being compressed into bits, much of the distance information is leaked out, and good detection results cannot be obtained. Therefore, modifying CenterNet to accept 16-bit data has the most favorable effect. Therefore, when using 16-bit data, it is possible to input the initial uncompressed depth information and provide full details of the AI model to improve the accuracy of its decisions.

The background server 2 receives the 3D depth image data captured by the imaging device 1 . Background server 2 includes at least one processor 21 and at least one computer-readable storage medium 22 .
Computer readable storage medium 22 stores at least one depth learning application 221 . The computer-readable storage medium 22 stores computer-readable commands, and when the plurality of processors 21 executes the computer-readable commands, the depth learning application 221 performs image detection and tracking based on the three-dimensional depth image data and the target environment. determine the number of people in, bed-on condition or/and non-bed-on condition.

The depth learning application 221 includes a depth training unit 2211 , an image receiving unit 2212 , an image detection and tracking unit 2213 and a state determination unit 2214 .
(1) The depth training unit 2211 performs model training on a plurality of field depth data stored in the background server 2 to construct a depth learning model.
(2) The image receiving unit 2212 receives 3D depth image data.
(3) The image detection and tracking unit 2213 is connected with the depth training unit 2211 and the image receiving unit 2212, and the image detection and tracking unit 2213 performs image detection and tracking on the 3D depth image data through the depth learning model. to detect the human head position and bed-shaped object position in the target environment and track the human head and bed-shaped object.
(4) The state determination unit 2214 is connected with the image detection and tracking unit 2213 to determine the number of people in the target environment according to the human head position, and to determine the overlapping state of the human head position and the bed-type object position. Based on this, the state on the bed is determined, and the width of the fall of the human head (incline acceleration from head tilt to stop) and fall width time (tilt time from head tilt to stop). Determine top condition.

The depth training unit 2211 performs model training with object detection algorithms and image detection and tracking on 3D depth image data with depth learning models. Object detection algorithm technologies are CenterNet, YOLO, SSD, DarkNet or MobileNet.

The depth training unit 2211 of the present invention collects and trains a large amount of various environmental data. The model trained by the present invention has the following model optimization processes (1) to (6).
(1) Collect depth data in various fields to increase data diversity.
(2) Marking and collecting the position in the screen where the human head is located in the depth data, and marking the position of the human head in the depth image, the human head that appears more than 65 to 70% in the screen. Marking and loss of marking due to having to mark the entire head can be reduced.
(3) model training.
(4) Test whether the trained model is more than 98% accurate in different environments. If not achievable, review and correct data markings.
(5) If the accuracy goal of the model is achieved, the algorithm is used in the actual field and tested for a week or more (after reliably marking and identifying the human head, the algorithm test against the actual field to determine the behavioral state of the human body (bed-on or non-bed-on)).
(6) Optimizing model data and behavioral algorithms to achieve optimal results when behaviors occurring in the process are unpredictable or undetectable.

The depth learning application 221 stores an image with a change in state each time it determines that a change in state has occurred, thus saving the cost and time required to store multiple times.

The actual steps of the deep learning application 221 will be described in (1)-(7) below based on the flowcharts shown in FIGS. 4A-4C.
(1) After image input (401), image detection (402) is started, and based on the number of people in the target environment, the target environment is empty (406) or alone (404). or multiple persons (405).
(2) Multi-Person State: Analyze Bed-On or/and Non-Bed-On State for One Person to Enter Caregiver-Care State and Accurately Determine Falls and Reduce Errors However, it is also possible to analyze and determine the bed state and/or the non-bed state for the states of a plurality of persons.
(3) Subsequently, image tracking (407) is started and overlap (408) is determined. That is, it is determined whether or not the human head position is within the range of the bed-shaped object position. If overlapped (i.e., within range), determine an on-bed condition; if not overlapped (i.e., not within range), determine a non-on-bed condition.
(4) If it is determined that they are overlapped, determine whether the human head position is higher than the first height (409), and if it is lower than the first height, the bed state is lying on the bed. (410) Determine (eg, lying on the bed). Conversely, if it is higher than the first height, it is determined whether the human head position is higher than the second height (411). If it is lower than the second height, the bed state is the upright state (412) (because the second height is close to the height of the upper body of the human body, if it is lower than the second height, the upper body is completely upright). (represents a state in which the If it is higher than the second height, it is determined that the state on the bed is sitting on the bed (413) (the upper body is upright because the height is higher than the second height) (for example, sitting on the bed). .
(5) If it is determined that there is no overlap, further determine whether there is a fall. It mainly determines the fall width of the human head according to the tilting speed and tilting time of the human head. Return to tracking (407) again.
(6) If the human head is tilted, the fall width of the human head is the acceleration reference value 106 cm/s, 107 cm/s, 108 cm/s, 109 cm/s, 110 cm/s, 111 cm/s, 112 cm/s, 113 cm/s, 114 cm/s, 115 cm/s, 116 cm/s, 117 cm/s, 118 cm/s s, 119 cm/s, 120 cm/s) 415 or not. If the decision is 'no' then determine 416 that the non-bed condition is a human intrusion condition (detects human intrusion into visibility range), if the decision is 'yes' then: The non-on-bed condition is determined to be a possible fall condition (417).
(7) After that, after judging that there is a possibility of falling, it is judged whether or not the stop time of the human head position has exceeded a certain period of time (418) (this can be set by oneself). If no, determine 419 that the non-in-bed condition is a minor fall condition. If the determination is "yes", determine 420 that the non-in-bed condition is a severe fall condition (eg, falling out of bed, losing balance when standing up, etc.).

The depth learning application differentiates between the first height and the second height based on the height data (the user can enter the height information in advance and set it in the system). The first height is 0.2-0.25 multiples of the height data. The second height is a 0.25-0.4 multiple of the height data.

After determining that the person is sitting on the bed, if it is detected that the human head position is not already within the range of the bed type object position within a certain period of time, is the state on the bed separated from the bed? Judging (judging that he is sitting on the edge of the bed and leaning away from the bed or preparing to get off the bed).

If it is detected that the head position of the human body has not changed within a certain period of time after determining that the person is lying on the bed, it is further determined whether the state on the bed is a stationary state.

The depth learning application determines whether the bed has moved or not moved within a period of time based on the position of the bed-type object in the target environment.

The background server 2 communicates with the electronic device 3 and sends the on-bed state and/or non-on-bed state determined by the depth learning application 221 to the electronic device 3 . Here, the electronic device 3 is a mobile phone, a tablet computer, a notebook computer, or a desktop computer. The posture state is transmitted to the electronic device 3, and several types of posture states will be described below in (1) to (6).

(1) Sitting on bed: Notify within 3 seconds after changing to sitting on bed, and if this state continues, notify once every 10 minutes.
(2) Away from bed state: Notify within 3 seconds after changing to Away from bed state, and notify once every 10 minutes if this state continues.
(3) Severe fall condition: Notify within 3 seconds after a serious fall condition occurs, and if this condition continues, notify once every 10 minutes.
(4) Stationary state: When the user lies on the bed and does not move at all for several tens of minutes, the user is notified, and when this state continues, the user is notified once every 10 minutes.
(5) Intruder presence state: When a person lies or sits on a bed, it detects and notifies that a person has entered and is approaching.
(6) Bed moved position: Issue an alert when the bed position is moved and notify the staff to return the bed to its original position. Until the bed is replaced, the detection system is turned off temporarily and alerted once every 10 minutes if the condition is not relieved.

The posture/action recognition system according to the present invention has the following advantages (1) and (2) compared with the conventional technology.
(1) When the patient leaves the bed, it is possible to monitor the action of leaving the bed, falls and other actions.
(2) Based on the position of the human head and the position of the bed-type object, determine the number of people in the target environment, the state on the bed, the state not on the bed, and the movement state of the bed position, and send the posture state to the electronic device for warning. and notification.

As can be appreciated by those skilled in the art, the preferred embodiments of the present invention have been disclosed above and are not intended to limit the invention in any way. Various changes and modifications may be made without departing from the spirit and scope of the invention. Therefore, the claims of the utility model registration of the present invention should be broadly interpreted including such changes and modifications.

1 imaging device 2 background server 3 electronic device 21 processor 22 computer readable storage medium 221 depth learning application 2211 depth training unit 2212 image receiving unit 2213 image detection and tracking unit 2214 state determination unit

Claims (3)

A posture/action identification system comprising at least one imaging device and at least one background server,
the imaging device capturing three-dimensional depth image data within a target environment;
The background server receives the three-dimensional depth image data captured by the imaging device,
The background server is implemented with a depth learning application, performs image detection and tracking based on the 3D depth image data, captures a human head position and a bed-type object position, and based on the human head position Judging the number of people in the target environment, judging the state on the bed based on the overlapping state of the position of the human head and the position of the bed-type object, and non-bed based on the drop width and drop width time of the human head Characterized by judging the upper state,
Posture/action recognition system. The depth learning application comprises a depth training unit, an image receiving unit, an image detection and tracking unit and a state determination unit, wherein the depth training unit performs model training on multiple field depth data stored in the background server. to build a depth learning model, the image receiving unit receives the three-dimensional depth image data, the image detection and tracking unit is connected with the depth training unit and the image reception unit, and performs image detection and tracking. The tracking unit includes a depth learning model, and performs image detection and tracking on the three-dimensional depth image data via the depth learning model to detect the human head position and the bed-shaped object position in the target environment. and tracking a human head and a bed-shaped object, wherein the state determination unit is connected with the image detection and tracking unit to determine the number of people in the target environment based on the human head position; The bed-on-bed state is determined based on the overlapping state of the head position and the bed-type object position, and the non-bed-on-bed state is determined based on the fall width and the fall width time of the human head. 2. The posture/action identification system according to 1. The three-dimensional depth image data is composed of three color channels of RGB or composed of grayscale image channels, and the number of input bits of the three-dimensional depth image data is 8 bits or 16 bits. Item 2. The posture/action recognition system according to item 1.

Images