TWI725803B - System for recognizing abnormal activity of human body using wearable electronic device and mixed reality technology - Google Patents

Abstract

The present invention discloses a system for recognizing abnormal activity of human body using wearable electronic device and mixed reality technology, comprising: a first wearable electronic device, a plurality of second wearable electronic device, at least one image capturing module, and a judgement module. This novel system is suitable for applying in any kinds of work environment, so as to monitor and determine whether a user (i.e., an operator or an employee) exhibits abnormal activities by referring a document of standard operating procedures (SOP), an operating manual and a document of safety operation standard (SOS), such that it is facilitated to reduce a happening rate of the user’s mishandling, thereby achieving the enhancement of work efficiency and productivity.

Description

Human body abnormal activity recognition system using wearable electronic device

The present invention relates to the technical field of application of wearable electronic devices, in particular to a human body abnormal activity recognition system using wearable electronic devices.

It is known that new employees of the company must be trained and passed the assessment before they can perform technical work. In the process of instructing and training new recruits, in addition to providing standard operating procedures (SOP) documents and/or paper and electronic files of the operation manual to new recruits, senior staff will also follow the instructions in the standard teaching manual. The teaching steps and methods provide relevant guidance and training for new recruits. After that, the new recruits can refer to the standard operating procedure documents and operation manuals to repeat the exercises step by step until they are familiar with the working procedures and machine operation methods.

However, for workers who have been familiar with standard operating procedures documents and operating manuals, and are familiar with working procedures and machine operation methods, unexpected emergencies and dangerous factors hidden in the actual working environment will still cause workers’ problems. Operation error or improper operation, serious cases may also lead to industrial safety accidents. In view of this, human factors engineering has been researched, proposed and applied to manage the interaction and reasonable combination of staff, machines and the environment to improve production efficiency. Unfortunately, the current application of human factors engineering has its limitations, which makes it unsuitable for various working environments. For example, when designing a production machine, the mechanical engineer must design the production machine as a Human-Machine System based on human engineering, so that the operator of the production machine can improve while effectively reducing the occurrence of errors. Work efficiency and productivity. However, as far as the reality is concerned, not every production machine, tool, workstation, work method, or work-related hardware equipment can be designed as a human machine system.

From the above description, it is necessary to redesign and develop a new type of human-machine system, human-machine device, or human-machine work that is different from the conventional one, so that this new system can be applied to various working environments, and then At the same time, it achieves multiple goals of reducing the operating error rate of operating (workers), effectively improving work efficiency, and promoting productivity. In view of this, the inventor of this case tried his best to research and invent, and finally completed the invention of a human body abnormal activity recognition system using a wearable electronic device.

The main purpose of the present invention is to provide a human body abnormal activity recognition system using a wearable electronic device, which includes: a first wearable electronic device, a plurality of second wearable electronic devices, at least one camera module, and a human body abnormality Activity judgment module. The human body abnormal activity recognition system of the present invention is suitable for application in various working environments for a user (for example: an operator or a worker) to use, and then according to standard operating procedures (SOP) documents, operating manuals, and / Or work safety specification documents to monitor and determine whether the user has occurred or performed abnormal human activities, by this way, at the same time to reduce the operation (work) personnel's operation error rate, effectively improve work efficiency, and promote productivity.

In the case of applying the human body abnormal activity recognition system 1 of the present invention, the back-end manager can instantly monitor the online operation of the operator to ensure that the operator is in accordance with the relevant regulations of the standard operating procedure document and/or operation manual Under the circumstances, perform its responsible work. At the same time, since the human body abnormal activity recognition system 1 of the present invention has the function of posture (action) estimation, it can predict whether the next action of the operator is correct. If the next action of the operator is predicted to be an abnormal human action, the back-end manager will receive a warning message to correct the action of the operator early to effectively prevent occupational injuries caused by operating errors.

In order to achieve the above objective, the present invention proposes an embodiment of the human body abnormal activity recognition system, which includes: A first wearable electronic device for wearing on the head of a user, and having a camera unit for capturing a first image of at least one object and a working environment, and for the user A head motion detection unit that performs a head motion detection on the head of the head, and then outputs a first object identification data, a first situation identification data, and a head behavior identification data; A plurality of second wearable electronic devices are used to be worn on the user's wrist, and they have a wrist motion detection unit for performing a wrist motion detection on the user's wrist, and then output A wrist behavior identification data; At least one camera module is used to perform a second image capture on the user, the at least one object and the working environment, and then output a second object identification data and a second contextual identification data, and the camera module The group has a posture estimation unit for generating a posture estimation data by performing posture estimation based on a user's body posture obtained by the second image capture; and A human body abnormal activity determination module, coupled to the first wearable electronic device, the plurality of second wearable electronic devices, and the photography module, and has: A data fusion unit for receiving the first object identification data, the first context identification data and the head behavior identification data sent by the first wearable electronic device, and the second wearable electronic device Wrist behavior identification data, and the data fusion unit simultaneously receives the second object identification data, the second context identification data, and the posture estimation data sent by the photography module, and then the first object identification data, the Perform a data fusion process on the first context recognition data, the head behavior recognition data, the wrist behavior recognition data, the second object recognition data, the second context recognition data, and the posture estimation data; and A decision unit, coupled to the data fusion unit, is used for judging whether the user has an abnormal human activity event based on a fusion data and at least one reference data sent by the data fusion unit.

In the aforementioned embodiment of the human body abnormal activity recognition system of the present invention, the first wearable electronic device can be any one of the following: a mixed reality (MR) helmet or smart glasses.

In the foregoing embodiment of the human body abnormal activity recognition system of the present invention, the second wearable electronic device can be any one of the following: a smart bracelet or a smart watch.

In the foregoing embodiment of the human body abnormal activity recognition system of the present invention, the human body abnormal activity judgment module further has a database for storing the reference data, and the reference data includes: standard operating procedure files, operation manuals, And work safety specification documents.

In the foregoing embodiment of the human body abnormal activity recognition system of the present invention, the head motion detection unit and the wrist motion detection unit are both an inertial sensor.

In the foregoing embodiment of the human body abnormal activity recognition system of the present invention, the human body abnormal activity determination module is a data processing device, and the data processing device can be any of the following: central control system, industrial computer, server Computer, desktop, notebook, tablet, or smart phone.

In the aforementioned embodiment of the human body abnormal activity recognition system of the present invention, the head movement detection includes head rotation angle detection, head rotation count detection, head tilt angle detection, and head position detection.

In the aforementioned embodiment of the human body abnormal activity recognition system of the present invention, the wrist movement detection includes wrist rotation angle detection, wrist rotation count detection, wrist tilt angle detection, and wrist tilt angle detection .

In a possible embodiment, the human body abnormal activity recognition system of the present invention further includes a third wearable electronic device, which is coupled to the human body abnormal activity judgment module and is used to be worn on the user's hand, and have: A hand motion detection unit for performing hand motion detection on the user's hand; and A hand behavior recognition unit is used for performing a hand behavior recognition processing based on a hand motion data obtained by the hand motion detection, so as to generate a hand behavior recognition data and send it to the abnormal human activity judgment module.

In the foregoing embodiment of the human body abnormal activity recognition system of the present invention, the hand motion detection includes fist detection, palm opening detection, finger motion detection, hand position detection, hand rotation angle detection, and hand motion detection. Detection of the number of rotations, hand tilt angle detection, and hand tilt angle detection.

In the foregoing embodiment of the human body abnormal activity recognition system of the present invention, the human body abnormal activity judgment module further includes: A data synchronization unit is coupled to the first wearable electronic device, the plurality of second wearable electronic devices, the photography module, and the third wearable electronic device to enable the first wearable electronic device, The plurality of second wearable electronic devices, the photography module, and the third wearable electronic device synchronously transmit the first object identification data, the first context identification data, the head behavior identification data, and the wrist behavior The identification data, the second object identification data, the second situation identification data, the posture estimation data, and the hand behavior identification data are sent to the data fusion unit.

In order to more clearly describe the human body abnormal activity recognition system using a wearable electronic device proposed by the present invention, the preferred embodiments of the present invention will be described in detail below in conjunction with the drawings.

FIG. 1 shows a first perspective view of a human body abnormal activity recognition system using a wearable electronic device according to the present invention. The present invention proposes a human body abnormal activity recognition system 1, which includes a variety of different types of wearable electronic devices. In particular, the human body abnormal activity recognition system 1 of the present invention is suitable for use in various working environments for a user 2 (for example: an operator or a worker) to use according to standard operating procedures (SOP) documents , Operation manual, and/or work safety specification documents to monitor and determine whether the user 2 has occurred or performed abnormal human activities, so as to reduce the operation error rate of the operation (work) personnel, effectively improve work efficiency, and promote The multiple purposes of productivity.

As shown in FIG. 1, the human body abnormal activity recognition system 1 of the present invention mainly includes: a first wearable electronic device 11, a plurality of second wearable electronic devices 12, at least one photographing module 13, and a human body abnormal activity judgment Module 14. Further, FIG. 2 shows a functional block diagram of the first wearable electronic device 11 and the abnormal human activity determination module 14, and FIG. 3 shows the second wearable electronic device 12, the photographing module 13 and the abnormal human body The function block diagram of the activity judgment module 14. It can be seen from FIGS. 1 and 2 that the first wearable electronic device 11 may be a mixed reality (MR) helmet for wearing on the head of the user 2 and it has: a camera unit 111. A head motion detection unit 112, a first object recognition unit HOR, a first context recognition unit HIS, and a head behavior recognition unit HHO.

Following the above description, the camera unit 111 performs a first image capture of at least one object and a working environment based on a first-person perspective of the user 2. Here, it must be explained that the object refers to the tools, parts, materials, equipment, machinery, or equipment that the user 2 needs to use when performing a specific task, and the working environment is It may be an office, clean room, or production line. On the other hand, the head motion detection unit 112 is an inertial sensor for performing head motion detection on the head of the user 2, wherein the head motion detection includes : Head rotation angle detection, head rotation count detection, head tilt angle detection, and head position detection. In more detail, the first object identification unit HOR is used for performing a first object identification process according to a first object data obtained by the first image capture to generate a first object identification data. In addition, the first context recognition unit HIS is used to perform a first context recognition process according to a second work environment data obtained by the first image capture to generate a first context recognition data. Furthermore, the head behavior recognition unit HHO is used to perform a head behavior recognition process according to a head motion data obtained by the motion detection to generate a first context recognition data.

It is supplemented that the first wearable electronic device 11 (for example: MR helmet) usually has a microprocessor. Therefore, in a feasible embodiment, the first object recognition unit HOR, the first context recognition unit HIS, and the head behavior recognition unit HHO can be edited to at least in the form of a library, variable or operand An application program is then created in the microprocessor of the first wearable electronic device 11. It is worth noting that although FIG. 1 shows that the first wearable electronic device 11 is an MR helmet, it does not limit the implementation of the first wearable electronic device 11 in this way. FIG. 4 shows a second perspective view of the human body abnormal activity recognition system using a wearable electronic device of the present invention. In FIG. 4, the first wearable electronic device 11 is in the form of smart glasses as another implementation aspect.

Continue to refer to Figure 1 and Figure 3. The plurality of second wearable electronic devices 12 are used to be worn on the wrist of the user 2 and are, for example, an inertial sensor of a smart bracelet. According to the design of the present invention, the second wearable electronic device 12 has a wrist motion detection unit 121 and a wrist behavior recognition unit WGAR, wherein the wrist motion detection unit 121 is used to detect the wrist of the user 2 Perform a wrist motion detection, and the wrist motion detection includes wrist rotation angle detection, wrist rotation count detection, wrist tilt angle detection, and wrist tilt angle detection. On the other hand, the wrist behavior recognition unit WGAR is used to perform a wrist behavior recognition process based on a wrist motion data obtained by the wrist motion detection, and finally generate a wrist behavior recognition data. It is worth noting that although FIG. 1 shows that the second wearable electronic device 12 is a smart bracelet, it does not limit the implementation of the second wearable electronic device 12 in this way. 5 shows a perspective view of another embodiment of the second wearable electronic device of the human body abnormal activity recognition system of the present invention. In FIG. 5, the second wearable electronic device 12 is in the form of a smart watch as another implementation aspect.

Similarly, the second wearable electronic device 12 (for example, a smart bracelet) usually has a microprocessor. Therefore, in a feasible embodiment, the wrist behavior recognition unit WGAR is edited as at least one application program in the form of a library, variable, or operand, and then is built on the second wearable electronic device 12 Among the microprocessors. As shown in FIGS. 1 and 3, the camera module 13 is used to perform a second image capture on the user 2, the at least one object, and the working environment. Simply put, the camera module 13 is used to simultaneously record the at least one object of the user 2 and the real-time image of the working environment. According to the design of the present invention, the photographing module 13 has a second object recognition unit FRO, a second context recognition unit FIS, and a posture estimation unit FPE. Wherein, the second object identification unit FOR is used for performing a second object identification process according to a second object data obtained by the second image capture to generate the second object identification data. In addition, the second context recognition unit FIS is used for performing a second context recognition process according to a second working environment data obtained by the second image capture to generate the second context recognition data. In particular, the posture estimation unit FPE is used for generating a posture estimation data according to a user's body posture obtained by the second image capture and a posture estimation.

It is supplemented that the photographing module 13 (for example, a digital camera or a smart phone) usually has a microprocessor. Therefore, in a feasible embodiment, the second object recognition unit FOR, the second context recognition unit FIS, and the posture estimation unit FPE are edited into at least one application in the form of a library, variable or operand The program is then built in the microprocessor of the photographing module 13. In more detail, the abnormal human activity determination module 14 is connected to the first wearable electronic device 11, the plurality of second wearable electronic devices 12, and the photographing module 13 for data transmission through a wireless connection. And it has: a data synchronization unit 140, a data fusion unit 141, and a decision unit 142. According to the design of the present invention, the data synchronization unit 140 is coupled to the first wearable electronic device 11, the plurality of second wearable electronic devices 12, and the photography module 13 to enable the first wearable electronic device 11. The plurality of second wearable electronic devices 12 and the photographing module 13 synchronously the first object identification data, the first context identification data, the head behavior identification data, the wrist behavior identification data, The second object identification data, the second context identification data, and the posture estimation data are sent to the data fusion unit 141.

Following the above description, the data fusion unit 141 is used to further identify the first object, the first situation identification data, the head behavior identification data, the wrist behavior identification data, the second object identification data, and the first object identification data. 2. The situation recognition data and the posture estimation data are subjected to a data fusion process. In addition, the decision unit 142 is coupled to the data fusion unit 141 for determining whether the user 2 has an abnormal human activity event based on a fusion data sent by the data fusion unit 141 and at least one reference data. It is supplemented that the abnormal human activity judgment module 14 is a data processing device. For example, FIG. 1 shows that the abnormal human activity judgment module 14 is a notebook computer. Therefore, the human body abnormal activity judgment module 14 has a storage unit for storing the reference data, and the reference data includes: standard operating procedure documents, operation manuals, and safety code documents. On the other hand, in a feasible embodiment, the data fusion unit 141 and the decision unit 142 are edited into at least one application program in the form of a library, variable or operand, and then established on the abnormal human body activity In a processor of the judging module 14. However, it should be known that the possible embodiments of the abnormal human activity judgment module 14 are not limited to notebook computers, but can also be other types of data processing devices, such as central control systems, industrial computers, and servers. Computer, desktop, tablet, or smart phone.

FIG. 6 shows a third perspective view of the human body abnormal activity recognition system using a wearable electronic device of the present invention. Comparing FIG. 1 with FIG. 6, it can be seen that in a feasible embodiment, the human body abnormal activity recognition system 1 of the present invention may further include a third wearable electronic device 15 for wearing on the hand of the user 2 and The human body abnormal activity judgment module 14 is coupled through a wireless connection and a wired connection. FIG. 7 shows a functional block diagram of the third wearable electronic device 15 and the abnormal human activity determination module 14. According to the design of the present invention, the third wearable electronic device 15 is a smart glove, and it has a hand motion detection unit 151 and a hand behavior recognition unit HGAR. The hand motion detection unit 151 is used to perform a hand motion detection on the hand of the user 2, and the hand behavior recognition unit HGAR is used to obtain a hand motion according to the hand motion detection. The data is subjected to a hand behavior recognition processing to generate a hand behavior recognition data to be sent to the abnormal human activity judgment module 14.

Following the above description, the hand motion detection includes fist detection, palm opening detection, finger motion detection, hand position detection, hand rotation angle detection, hand rotation count detection, hand tilt angle Detection, and hand tilt angle detection. Moreover, it can be inferred that the data synchronization unit 140 of the abnormal human activity judgment module 14 is simultaneously coupled to the first wearable electronic device 11, the plurality of second wearable electronic devices 12, the photographing module 13, and the The third wearable electronic device 15 is used to enable the first wearable electronic device 11, the plurality of second wearable electronic devices 12, the photography module 13, and the third wearable electronic device 15 to synchronize the The first object identification data, the first situation identification data, the head behavior identification data, the wrist behavior identification data, the second object identification data, the second situation identification data, the posture estimation data, and the hand The behavior identification data is sent to the data fusion unit 141.

In this way, the above system has completely and clearly explained a human body abnormal activity recognition system using a wearable electronic device of the present invention; and, from the above, it can be seen that the present invention has the following advantages:

(1) The present invention uses a first wearable electronic device 11, a plurality of second wearable electronic devices 12, at least one camera module 13, and a human body abnormal activity judgment module 14 to form a human body abnormal activity recognition system 1. The human body abnormal activity recognition system 1 of the present invention is suitable for application in various working environments for a user 2 (for example: an operator or a worker) to use according to standard operating procedures (SOP) documents and operating manuals , And/or work safety specification documents to monitor and determine whether the user 2 has occurred or performed abnormal human activities, so as to reduce the operation error rate of operators (workers), effectively improve work efficiency, and promote productivity. purpose.

(2) With the application of the human body abnormal activity recognition system 1 of the present invention, the back-end manager can instantly monitor the online operation actions of the operators to ensure that the operators are in accordance with the standard operating procedure documents and/or operating manuals. Perform its responsible work under relevant regulations. At the same time, since the human body abnormal activity recognition system 1 of the present invention has the function of posture (action) estimation, it can predict whether the next action of the operator is correct. If the next action of the operator is predicted to be an abnormal human action, the back-end manager will receive a warning message to correct the action of the operator early to effectively prevent occupational injuries caused by operating errors.

It must be emphasized that the above detailed description is a specific description of possible embodiments of the present invention, but this embodiment is not intended to limit the patent scope of the present invention. Any equivalent implementation or modification that does not deviate from the technical spirit of the present invention, All should be included in the patent scope of this case.

>The invention> 1: Human body abnormal activity recognition system 11: The first wearable electronic device 111: camera unit 112: head motion detection unit HOR: The first object recognition unit HIS: The first situation recognition unit HHO: Head Behavior Recognition Unit 12: The second wearable electronic device 121: Wrist motion detection unit WGAR: Wrist Behavior Recognition Unit 13: Photography module FRO: Second Object Recognition Unit FIS: Second Situation Recognition Unit FPE: Posture Estimation Unit 14: Abnormal human activity judgment module 140: data synchronization unit 141: Data Fusion Unit 142: Decision Unit 15: The third wearable electronic device 151: Hand motion detection unit HGAR: Hand Behavior Recognition Unit 2: User

Figure 1 shows a first perspective view of a human body abnormal activity recognition system using a wearable electronic device according to the present invention; 2 shows a functional block diagram of the first wearable electronic device and the abnormal human activity judgment module of the human body abnormal activity recognition system of the present invention; 3 shows a functional block diagram of the second wearable electronic device, the photographing module, and the abnormal human activity judging module of the human body abnormal activity recognition system of the present invention; Figure 4 shows a second perspective view of the human body abnormal activity recognition system of the present invention; FIG. 5 shows a perspective view of another embodiment of the second wearable electronic device; Figure 6 shows a third perspective view of the human body abnormal activity recognition system of the present invention; and FIG. 7 shows a functional block diagram of the third wearable electronic device and the abnormal human activity judging module of the human body abnormal activity recognition system of the present invention.

1: Human body abnormal activity recognition system

11: The first wearable electronic device

12: The second wearable electronic device

13: Photography module

14: Abnormal human activity judgment module

2: User

Claims (10)

A recognition system for abnormal human activities, including: A first wearable electronic device for wearing on the head of a user, and having a camera unit for capturing a first image of at least one object and a working environment, and for the user A head motion detection unit that performs a head motion detection on the head of the head, and then outputs a first object identification data, a first situation identification data, and a head behavior identification data; A plurality of second wearable electronic devices are used to be worn on the user's wrist, and they have a wrist motion detection unit for performing a wrist motion detection on the user's wrist, and then output A wrist behavior identification data; At least one camera module is used to perform a second image capture on the user, the at least one object and the working environment, and then output a second object identification data and a second contextual identification data, and the camera module The group has a posture estimation unit for generating a posture estimation data by performing posture estimation based on a user's body posture obtained by the second image capture; and A human body abnormal activity determination module, coupled to the first wearable electronic device, the plurality of second wearable electronic devices, and the photography module, and has: A data fusion unit for receiving the first object identification data, the first context identification data and the head behavior identification data sent by the first wearable electronic device, and the second wearable electronic device Wrist behavior identification data, and the data fusion unit simultaneously receives the second object identification data, the second context identification data, and the posture estimation data sent by the photography module, and then the first object identification data, the Perform a data fusion process on the first context recognition data, the head behavior recognition data, the wrist behavior recognition data, the second object recognition data, the second context recognition data, and the posture estimation data; and A decision unit, coupled to the data fusion unit, is used for judging whether the user has an abnormal human activity event based on a fusion data and at least one reference data sent by the data fusion unit. The human body abnormal activity recognition system according to claim 1, wherein the abnormal human activity judgment module is a data processing device, and the data fusion unit and the decision unit are implemented in the form of a function library, a variable, or an operand It is edited into at least one application program, and then is built in a processor of the data processing device. The human body abnormal activity recognition system according to claim 1, wherein the abnormal human activity judgment module further has a storage unit for storing the reference data, and the reference data includes: standard operating procedure documents, operation manuals, And work safety specification documents. According to the human body abnormal activity recognition system described in claim 1, the first wearable electronic device further has: A first object identification unit for performing a first object identification process based on a piece of first object data obtained by the first image capture to generate the first object identification data; A first context recognition unit for performing a first context recognition process based on a second work environment data obtained by the first image capture to generate the first context recognition data; and A head behavior recognition unit is used for performing a head behavior recognition process according to a head motion data obtained by the motion detection to generate the first situation recognition data. The human body abnormal activity recognition system according to claim 1, wherein the second wearable electronic device further has: A wrist behavior recognition unit is used to perform a wrist behavior recognition process based on a wrist motion data obtained by the wrist motion detection to generate the wrist behavior recognition data. The human body abnormal activity recognition system according to claim 4, wherein the first object recognition unit, the first context recognition unit, and the head behavior recognition unit are edited in the form of a library, variable or operand It is at least one application program, which is then built in a microprocessor of the first wearable electronic device. The human body abnormal activity recognition system according to claim 4, wherein the photography module further has: A second object identification unit for performing a second object identification process based on a second object data obtained by the second image capture to generate the second object identification data; and A second context recognition unit is used for performing a second context recognition process according to a second working environment data obtained by the second image capture to generate the second context recognition data. The human body abnormal activity recognition system according to claim 7, wherein the second object recognition unit, the second context recognition unit, and the posture estimation unit are edited in the form of a library, variable, or operand as At least one application program is then built in a microprocessor of the camera module. The human body abnormal activity recognition system according to claim 1, further comprising a third wearable electronic device, coupled to the human body abnormal activity judgment module, and used to be worn on the user's hand, and has: A hand motion detection unit for performing hand motion detection on the user's hand; and A hand behavior recognition unit is used for performing a hand behavior recognition processing based on a hand motion data obtained by the hand motion detection, so as to generate a hand behavior recognition data and send it to the abnormal human activity judgment module. The human body abnormal activity recognition system according to claim 9, wherein the human body abnormal activity judgment module further includes: A data synchronization unit is coupled to the first wearable electronic device, the plurality of second wearable electronic devices, the photography module, and the third wearable electronic device to enable the first wearable electronic device, The plurality of second wearable electronic devices, the photography module, and the third wearable electronic device synchronize the first object identification data, the first context identification data, the head behavior identification data, and the wrist The behavior recognition data, the second object recognition data, the second context recognition data, the gesture estimation data, and the hand behavior recognition data are sent to the data fusion unit.

Images