TWI755950B - Action recognition method and system thereof - Google Patents

Abstract

The present invention provides an action recognition method and system thereof. The action recognition method comprises: capturing a 2D image and a depth image at the same time, extracting an 2D information of the human skeleton points from the 2D image and correcting it, mapping the 2D information of the human skeleton points to the depth image to obtain the corresponding depth information with respect to the 2D information of the human skeleton points and combining the corrected 2D information of the human skeleton points and the depth information to obtain the 3D information of the human skeleton points, and finally recognizing an action from a set of 3D information of the human skeleton points during a period of time by a matching model.

Description

Action recognition method and system thereof

The present invention relates to an action recognition method and system, in particular to a multimodal image integration and simulation action recognition method and system.

Human Action Recognition (HAR) is a very popular research topic in recent years, and a lot of methods and technologies have been developed in the field of situational awareness, motion monitoring, and elderly care. Among them, the human skeleton point positioning technology in 2D images has become mature, and real-time 2D RGB images (red, green and blue images) or IR images (infrared images) can be used to identify and locate the head, torso, Upper and lower limbs, and then judge the activity state of human beings. However, in the recognition of some human activities, only two-dimensional skeleton point information is often used to distinguish. For example, the projections of the skeleton points of some actions on the plane overlap in many places, so it is impossible to identify and distinguish.

Therefore, as shown in Figure 1, higher-accuracy human activity recognition still often relies on the coordinate information of the 3D point cloud of the human body. The amount of 3D point cloud coordinate information obtained by using the 3D sensor is extremely large. Therefore, if the resolution of the 3D sensor is too high, it will take too much resources and time to calculate the human skeleton point location map, and if the resolution is too low, it may The correct skeleton point cannot be recognized due to background noise, which reduces the accuracy of action recognition. therefore, There is an urgent need for a real-time and highly accurate action recognition method and system.

The present invention provides a motion recognition method, comprising: capturing a two-dimensional color image or a two-dimensional infrared image and a corresponding depth image at a time point; extracting two of the two-dimensional color image or the two-dimensional infrared image 2D human skeleton point information; map the 2D human skeleton point information to the depth image to obtain depth information corresponding to the 2D human skeleton point information; use a size-depth parameter and a distortion model to correct the 2D human skeleton point information; combining the corrected 2D human skeleton point information with the depth information to obtain a 3D human skeleton point information; and using a matching model to identify an action for a series of the 3D human skeleton point information over a period of time.

The present invention further provides a motion recognition system, comprising: an image capturing device for capturing a two-dimensional color image or a two-dimensional infrared image at a time point; a depth image capturing device for capturing the time a depth image corresponding to one of the points; a memory for storing a size-depth parameter, a distortion model and a matching model; and a processor telecommunicationly connected to the image capture device, the depth image capture device and the The memory, the processor includes: an input module for receiving the two-dimensional color image or the two-dimensional infrared image and the corresponding depth image; a storage module for the two-dimensional color image or the two The 2D infrared image and the corresponding depth image are stored in the memory; a skeleton point calculation module is used to extract the 2D color image or the 2D human skeleton point information in the 2D infrared image, and use the size- The depth parameter and the distortion model correct the two-dimensional human skeleton point information; a mapping module is used to map the two-dimensional human skeleton point information to the depth image to obtain a depth data corresponding to the two-dimensional human skeleton point information information, and combining the corrected two-dimensional human skeleton point information and the depth information to obtain a three-dimensional human skeleton point information; and a motion recognition module using a matching model for a series of the three-dimensional human skeleton point information over a period of time Identify an action.

In some specific embodiments, an output module further includes an output module to issue a prompt signal when identifying the action.

In some embodiments, the matching model is a classification model parameter established by a neural network-like deep learning architecture.

In some embodiments, the distortion model is used to correct the distance between the pixel coordinate position of the two-dimensional human skeleton point and the image distortion center.

In some embodiments, the memory further stores a set of displacement parameters, and the depth image is first calibrated using the set of displacement parameters.

The motion recognition method and system provided by the present invention can solve the problems of time-consuming calculation of three-dimensional skeleton points of the human body and being easily affected by device resolution or noise, and propose a multi-modality (multi-modality) image integration, which can quickly and accurately simulate The method and system for three-dimensional skeleton point information can be applied to various real-time human activity recognition situations, such as fall situation detection.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

As used herein, the articles "a," "an," and "any" refer to grammatical items of one or more than one (ie, at least one) item. For example, "an element" means an element or an element.

As used herein, the terms "about", "approximately" or "approximately" mean substantially the stated value or range is within 20%, preferably within 10%, and more preferably within 5% Inside. Numerical quantities provided herein are approximations intended to be inferred if the terms "about," "approximately," or "approximately" were not used.

10: Motion Recognition System

11: Image capture device

12: Depth image capture device

13: Memory

14: Processor

141: Input module

142: Storage Module

143: Skeleton point calculation module

144: Mapping Module

145: Motion Recognition Module

146: Output module

S10: Step 10

S20: Step 20

S30: Step 30

S40: Step 40

S50: Step 50

S60: Step 60

FIG. 1 is a localization diagram of human skeleton points calculated by capturing human motions using a 3D sensor.

FIG. 2 is a block diagram of a motion recognition system according to an embodiment of the present invention.

FIG. 3 is a flowchart of a motion recognition method according to an embodiment of the present invention.

FIG. 4A is a schematic diagram of a grayscale of a skeleton point of a non-falling color image according to an embodiment of the present invention.

FIG. 4B is a schematic diagram of a grayscale of a skeleton point of a color image of a fall dynamic process according to an embodiment of the present invention.

FIG. 5A is a grayscale schematic diagram of a skeleton point in a non-falling depth image according to an embodiment of the present invention.

FIG. 5B is a grayscale schematic diagram of a skeleton point of a depth image of a dynamic fall process according to an embodiment of the present invention.

FIG. 6A is a grayscale schematic diagram of coordinate mapping of close-range skeleton points according to an embodiment of the present invention.

FIG. 6B is a grayscale schematic diagram of a long-distance skeleton point coordinate mapping according to an embodiment of the present invention.

FIG. 7 is a grayscale schematic diagram of motion recognition according to an embodiment of the present invention.

Other technical contents, features and effects of the present invention will be clearly presented in the following detailed description of the preferred embodiments with reference to the drawings.

As shown in FIG. 2 , an embodiment of the present invention provides a motion recognition system 10 , including: an image capturing device 11 , a depth image capturing device 12 , a memory 13 and a processor 14 . The processor 14 includes an input module 141 , a storage module 142 , a skeleton point calculation module 143 , a mapping module 144 , and a motion recognition module 145 . The motion recognition system 10 may further include an output module 146 .

As shown in FIG. 3 , an embodiment of the present invention provides a motion recognition method, which includes: capturing a two-dimensional color image or a two-dimensional infrared image and a corresponding depth image at a time point (step S10 ); extracting the two 2D color image or 2D human skeleton point information in the 2D infrared image (step S20); map the 2D human skeleton point information to the depth image to obtain a depth information corresponding to the 2D human skeleton point information ( Step S30); use a size-depth parameter and a distortion model to correct the two-dimensional human skeleton point information (step S40); combine the corrected two-dimensional human skeleton point information and the depth information to obtain a three-dimensional human skeleton point information (Step S50 ); and use a matching model to identify an action for a series of the three-dimensional human skeleton point information over a period of time (Step S60 ).

Please refer to FIG. 2 and FIG. 3 at the same time to understand the embodiment of the present invention. The image capturing device 11 is used for capturing a two-dimensional color image or a two-dimensional infrared image at a time point. The two-dimensional color image can be a flat RGB color image, such as a photo captured by a common camera or a frame of a video captured by a video recorder. Each pixel in the two-dimensional color image records a color information, which can be the content matrix information of red, green and blue. Two-dimensional infrared images can be Flat gray-scale images under near-infrared illumination are commonly used for identification or detection at night, and gray-scale images with good resolution can still be captured in the case of insufficient lighting. Each pixel in the 2D infrared image represents the infrared intensity detected by the infrared sensor.

The depth image capturing device 12 is used to capture a depth image corresponding to one of the time points, which may be a time of flight (TOF) sensor or a depth camera (eg, Intel RealSense). In order to map each other, it is necessary to obtain the corresponding depth image at the same time when capturing the two-dimensional color image or the two-dimensional infrared image. The depth image is also a two-dimensional image, but each pixel in the two-dimensional image represents the distance between the plane where the object captured by the pixel is located and the sensor.

The memory 13 is used to store a matching model for identifying different actions. In the embodiment of the present invention, taking the recognition of a falling motion as an example, the height of the sensing device is 2 meters, which can be the image capture device and the depth image capture device of the embodiment of the present invention, and a total of 60,000 images are taken with a resolution of 620*350 The continuous frame picture of the falling dynamic and the non-falling dynamic continuous frame picture are about half and half. Sampling the falling and non-falling dynamic continuous frame sequence (sequence), perform two-dimensional human skeleton point calculation for each frame picture in the sequence, and calculate the simulated three-dimensional skeleton point coordinates in combination with the corresponding depth image. Combining the three-dimensional skeleton point coordinates of each frame in the whole sequence, a four-dimensional dynamic feature sequence is obtained as the input feature of action recognition. The dynamic coordinate point sequence of the three-dimensional human skeleton point in time series is an important feature of action recognition, and a neural network-like deep learning architecture can be used, such as long short-term memory model (Long Short-Term Memory, LSTM) or convolutional neural network. Road (CNN) for deep learning to build a matching model that can recognize the different dynamic activities of the subjects.

The processor 14 is telecommunicationly connected to the image capture device 11, the depth image capture device set and the memory 12 . The image capturing device 11 and the depth image capturing device 12 capture a two-dimensional color image or two-dimensional infrared image and a corresponding depth image at a time point (step S10 ), and then transmit them to the processor 14 by wire or wirelessly. The input module 141 is used for receiving the two-dimensional color image or the two-dimensional infrared image and the corresponding depth image. In order to facilitate subsequent applications, the storage module 142 can store the two-dimensional color image or the two-dimensional infrared image and the corresponding depth image in the memory 13 for retrieval and use at any time.

Please refer to FIG. 4A and FIG. 4B at the same time, which are grayscale schematic diagrams of skeleton points of a 2D color image under non-falling and falling dynamic process conditions, although the embodiment of the present invention uses a 2D color image as an example and a grayscale schematic diagram However, the system and method of the embodiments of the present invention are not limited to using two-dimensional color images, and the two-dimensional infrared images can also have the same effect as grayscale images. The skeleton point calculation module 143 is used for extracting the 2D human skeleton point information in the 2D color image or the 2D infrared image (step S20). To identify 2D human skeleton point information in 2D color image or 2D infrared image, the parallel convolutional network architecture can be used to detect the confidence map of the joint point position and obtain the joint affine field (Part Affinity Fields) In order to describe the degree of connection between the joints, two features are integrated to predict each limb segment, and finally the two-dimensional human skeleton point information is obtained.

The 2D human skeleton point information is a data list containing 2D coordinates, which can indicate the pixel position of the real human skeleton point corresponding to the 2D color image or the 2D infrared image, which is the real plane mapped to the 2D color image. The relative position, the common form can be the pixel position of 18 skeleton points, that is, a 2x18 matrix. For example, the center point on the head in the non-falling image of FIG. 4A represents the pixel position where the nose is located in the two-dimensional color image as (361, 88).

Please refer to FIG. 5A and FIG. 5B at the same time, which are the depth images in the non-falling and falling motions The grayscale diagram of the skeleton point in the state of the state process. The key point of the embodiments of the present invention is to quickly obtain 3D human skeleton point information, first obtain plane human skeleton point information by using 2D color image or 2D infrared image, and then combine with depth image to form 3D human skeleton point information. Therefore, the two-dimensional color image or the two-dimensional infrared image and the depth image are firstly corresponded, and the depth information is obtained from the corresponding depth image. The mapping module 144 is used for mapping the 2D human skeleton point information to the depth image to obtain depth information corresponding to the 2D human skeleton point information (step S30 ). When mapping the human skeleton point information in the 2D color image or the 2D infrared image to the depth image, the corresponding pixel position of the human skeleton point in the 2D color image or the 2D infrared image can be obtained on the depth image to obtain the corresponding The value is the distance between the plane where the human skeleton point captured by the pixel is located and the sensor, that is, the depth information.

Although the 2D color image or 2D infrared image and depth image are captured at the same time, there will be a slight distance difference between the two image capture devices, or there will be different field sizes on the captured images. In order to improve the mapping A simple registration correction can be performed before the image capture device is used to construct a set of displacement parameters for subsequent correction of the depth image, so that the field of view size of the depth image, the image capture position and the two-dimensional color Image or 2D infrared image is the same. Using a calibration plate or a measured object, compare the corresponding position coordinates in the 2D color image or 2D infrared image and the depth image, and generate registration correction through image warping and reverse mapping. After the depth image, the pixel position of the same feature in the depth image is consistent with the pixel position of the feature in the two-dimensional color image or the two-dimensional infrared image. The set of displacement parameters of the depth image after the registration correction can be applied to the subsequent depth image correction, and can be stored in the memory 13 . This set of displacement parameter examples can be the displacements of several important calibration points, and the rest of the coordinates can be adjusted by interpolation to save computing time.

As shown in FIG. 6A and FIG. 6B , for the same subject and the image capturing device, when the subject is at different distances, the projected two-dimensional color images have different sizes. The closer to the image capturing device, the larger the captured person ( FIG. 6A ), and the farther away from the image capturing device, the smaller the captured character ( FIG. 6B ). Even for the same subject, the distances between the human skeleton points are inconsistent due to the inconsistent projection size, which will lead to subsequent motion recognition errors. The two-dimensional human skeleton point restores a consistent scale coordinate space according to its corresponding depth information, so as to simulate and reconstruct the three-dimensional Cartesian coordinate system position of the human skeleton point. Since such restoration only needs to be performed on the extracted two-dimensional human skeleton point information, a lot of time and resources can be saved.

A size-depth parameter is obtained by measuring the calibration plate or the corresponding image size of the same object at different positions, and then calculating the corresponding scale scale of the calibration plate or the measured object at different distances by means of linear interpolation. The size-depth parameters can be stored in the memory 13, and the skeleton point calculation module 143 can use the size-depth parameters to correct the 2D human skeleton point information (step S40), that is, first obtain the 2D color image or the 2D infrared image. The depth information corresponding to the two-dimensional human skeleton point information, and the corresponding scale scale calculated by the size-depth parameter is used to correct and restore the two-dimensional human skeleton point information, so as to adjust the size of the human skeleton of different depths to the same scale.

However, due to the curvature of the mirror surface of the lens of each image capture device, distortion on the distant image will be caused. Even if the distortion effect in the two-dimensional image is not obvious, the distortion of the image will be amplified in the scale restoration of the corresponding depth, causing the 3D skeleton point of the human body to have limb asymmetry after the scale restoration, especially in the distance. When the image capture device is farther away or deviated from the shooting center point, the distortion and distortion after the coordinate restoration will be more serious.

，其中

、

為校正後的點座標，x、y為原始影像 點座標，x _c 、y _c 為畸變中心點。L(r)為畸變模型，

，r為原始 座標距離畸變中心點的距離。對二維色彩影像或二維紅外線影像進行畸變校正還原。畸變模型可以儲存在記憶體13，骨架點計算模組143可以使用畸變模型校正二維人體骨架點資訊(步驟S40)。接著，映射模組144結合經校正之該二維人體骨架點資訊與該深度資訊以計算得到一三維人體骨架點資訊(步驟S50)，此三維人體骨架點資訊也就非常接近真實的骨架點空間位置。 To solve this problem, it is necessary to perform image distortion correction and restoration for different photographing devices. Use the calibration plate to capture multiple 2D color images or 2D infrared images, calculate the internal curvature parameter k of the lens, and use the division distortion model L(r) to correct and restore by reverse mapping.

,in

,

are the corrected point coordinates, x , y are the original image point coordinates, x _c , y _c are the distortion center points. L(r) is the distortion model,

, r is the distance from the original coordinate to the center point of the distortion. Distortion correction and restoration of 2D color images or 2D infrared images. The distortion model can be stored in the memory 13, and the skeleton point calculation module 143 can use the distortion model to correct the two-dimensional human skeleton point information (step S40). Next, the mapping module 144 combines the corrected 2D human skeleton point information and the depth information to calculate a 3D human skeleton point information (step S50 ). The 3D human skeleton point information is also very close to the real skeleton point space Location.

As shown in FIG. 7 , the embodiment of the present invention can be applied in the field of fall detection, but is not limited to the field of falls, and can also be used in the field of sports training and the like. The motion recognition module 145 uses a matching model to recognize a motion for a series of the three-dimensional human skeleton point information over a period of time (step S60 ). The 3D human skeleton point information can be a four-dimensional matrix, that is, the 3D human skeleton point information for a continuous period of time. The commonly used time length can be 1 to 2 seconds, more preferably 1.5 seconds, so as to achieve real-time. Action recognition. When the motion recognition schematic diagram is marked in the depth image, false colors can be used to represent different depth information, for example, red represents the distance from the image capture device, and blue represents the distance from the image capture device. The matching model is based on the parameters of the behavior classification model established by the deep learning framework, and is used to calculate which motion of the current subject's dynamic action matches the action in the model, so as to determine and identify an action, such as a fall.

The motion recognition system 10 according to the embodiment of the present invention further includes an output module 146 to issue a prompt signal when the motion is recognized. In the field of fall detection, the alert signal can trigger an alarm bell or a phone call to notify family members or police units. Figure 7 Left column shows different falls In the detection area, the upper right column displays a fall reminder signal, and the lower right column displays the detected fall screen.

The embodiment of the present invention uses RGB two-dimensional color images or two-dimensional infrared images to extract two-dimensional human skeleton point information, and combines depth information to quickly simulate a series of three-dimensional human skeleton point coordinates for a period of time as input features for behavior recognition, not only Compared with the accuracy of the 2D human skeleton points, it saves resources and computing time compared to the 3D human skeleton points measured by the 3D sensor. If it is used as a fall detection system for the elderly in real-time long-term care, it can solve the problem that many plane skeleton points cannot be accurately identified due to the overlapping of skeleton points on the plane in the action/behavior.

S10: Step 10

S20: Step 20

S30: Step 30

S40: Step 40

S50: Step 50

S60: Step 60

Claims (8)

A motion recognition method, comprising: capturing a two-dimensional color image or a two-dimensional infrared image and a corresponding depth image at a time point; extracting a two-dimensional human skeleton in the two-dimensional color image or the two-dimensional infrared image point information; map the 2D human skeleton point information to the depth image to obtain depth information corresponding to the 2D human skeleton point information; use a size-depth parameter and a distortion model to correct the 2D human skeleton point information, wherein the distortion model is used to correct the distance between the pixel coordinate position of the 2D human skeleton point and the image distortion center; combine the corrected 2D human skeleton point information and the depth information to obtain a 3D human skeleton point information; and A motion is identified for a series of the 3D human skeleton point information over a period of time using a matching model. The action recognition method of claim 1, further comprising: issuing a prompt signal when the action is recognized. The action recognition method of claim 1, wherein the matching model is a classification model parameter established by a neural network-like deep learning framework. The motion recognition method of claim 1, wherein the depth image is first corrected with a set of displacement parameters. An action recognition system comprising: an image capturing device for capturing a two-dimensional color image or a two-dimensional infrared image at a time point; a depth image capturing device for capturing a corresponding depth image at the time point; a memory a body for storing a size-depth parameter, a distortion model and a matching model; and a processor telecommunicationly connected to the image capture device, the depth image capture device and the memory, the processor comprising: an input a module for receiving the two-dimensional color image or the two-dimensional infrared image and the corresponding depth image; a storage module for storing the two-dimensional color image or the two-dimensional infrared image and the corresponding depth image to the memory; a skeleton point calculation module for extracting 2D human skeleton point information in the 2D color image or the 2D infrared image, and correcting the 2D human body using the size-depth parameter and the distortion model Skeleton point information, wherein the distortion model is used to correct the distance between the pixel coordinate position of the 2D human skeleton point and the image distortion center; a mapping module is used to map the 2D human skeleton point information to the depth image to obtain depth information corresponding to the two-dimensional human skeleton point information, and combining the corrected two-dimensional human skeleton point information and the depth information to obtain a three-dimensional human skeleton point information; and a motion recognition module using a matching model An action is recognized for a series of the three-dimensional human skeleton point information over a period of time. The motion recognition system of claim 5 further comprises: an output module that sends out a prompt signal when the motion is recognized. The action recognition system of claim 5, wherein the matching model is a neural network-like The classification model parameters established by the deep learning architecture. The motion recognition system of claim 5, wherein the memory further stores a set of displacement parameters, and the depth image is first corrected using the set of displacement parameters.

Images