EP4634878A1

EP4634878A1 - A computer-implemented method for action recognition and application for vehicle occupant or outside vehicle agent monitoring

Info

Publication number: EP4634878A1
Application number: EP23824902.3A
Authority: EP
Inventors: Zhengyu Li; Heng Guo; Lap Pui Chau; Cheen Hau TAN; Xiaoxi MA; Kim Hui YAP
Original assignee: Continental Automotive Technologies GmbH; Nanyang Technological University
Current assignee: Aumovio Germany GmbH; Nanyang Technological University
Priority date: 2022-12-14
Filing date: 2023-12-07
Publication date: 2025-10-22
Also published as: DE102022133342A1; WO2024126236A1

Abstract

In order to improve action recognition, a computer-implemented method for recognizing an action performed by an agent (10) in a sequence of frames (12) is proposed. The action is recognized by performing object detection on each frame (46) to obtain an object position and an object class of an object (14) present in the frames (46); generating a sequence of object class heatmaps (40) for each object class; extracting from each frame (46) keypoints (18) and limbs (20) of a skeleton (16) of the agent (10) and generating at least one sequence of key feature heatmaps (22, 30); stacking each key feature heatmap (22, 30) and each object class heatmap (41) to obtain a sequence of combined heatmaps (45); and feeding the sequence of combined heatmaps (45) into a machine learning model (48) that is configured as an action classifier and determining the action performed by the agent (10) based on the sequence of combined heatmaps (45). In contrast to known methods, the object class heatmaps (40) are augmented with the skeleton (16) of the agent (10)

Description

DESCRIPTION

A computer-implemented method for action recognition and application for vehicle occupant or outside vehicle agent monitoring

TECHNICAL FIELD

The invention relates to computer vision tasks, specifically recognition of actions taken by an agent. The invention further relates to applying the action recognition to monitoring a vehicle occupant inside the vehicle or an agent outside the vehicle.

BACKGROUND

Reference is made to the following documents:

[1] Duan, Haodong, Yue Zhao, Kai Chen, Dian Shao, Dahua Lin, and Bo Dai. "Revisiting Skeleton-based Action Recognition." arXiv preprint arXiv:2104.13586 (2021 ).

[2] Shahroudy, Amir, Jun Liu, Tian-Tsong Ng, and Gang Wang. "Ntu rgb+ d: A large scale dataset for 3d human activity analysis." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1010-1019. 2016.

[3] Liu, Ziyu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang, and Wanli Ouyang. "Disentangling and unifying graph convolutions for skeleton-based action recognition." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 143-152. 2020.

[4] Yan, Sijie, Yuanjun Xiong, and Dahua Lin. "Spatial temporal graph convolutional networks for skeleton-based action recognition." In Thirty-second AAAI conference on artificial intelligence. 2018.

[5] Chen, Yuxin, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. "Channel-wise topology refinement graph convolution for skeleton-based action recognition." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13359-13368. 2021.

[6] Shi, Lei, Yifan Zhang, Jian Cheng, and Hanging Lu. "Skeleton-based action recognition with multi-stream adaptive graph convolutional networks." IEEE Transactions on Image Processing 29 (2020): 9532-9545.

[7] Zhu, Jiagang, Wei Zou, Liang Xu, Yiming Hu, Zheng Zhu, Manyu Chang, Junjie Huang, Guan Huang, and Dalong Du. "Action machine: Rethinking action recognition in trimmed videos." arXiv preprint arXiv: 1812.05770 (2018).

[8] Carreira, Joao, and Andrew Zisserman. "Quo vadis, action recognition? a new model and the kinetics dataset." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308. 2017.

[9] Feichtenhofer, Christoph, Axel Pinz, and Andrew Zisserman. "Convolutional two- stream network fusion for video action recognition." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1933-1941 . 2016.

[10] Behera, Ardhendu, Alexander Keidel, and Bappaditya Debnath. "Context-driven multi-stream LSTM (M-LSTM) for recognizing fine-grained activity of drivers."

In German Conference on Pattern Recognition, pp. 298-314. Springer, Cham, 2018.

[11] Ji, Shuiwang, Wei Xu, Ming Yang, and Kai Yu. "3D convolutional neural networks for human action recognition." IEEE transactions on pattern analysis and machine intelligence 35, no. 1 (2012): 221-231.

[12] Choutas, Vasileios, Philippe Weinzaepfel, Jerome Revaud, and Cordelia Schmid. "Potion: Pose motion representation for action recognition." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7024- 7033. 2018. [13] Liu, Mengyuan, and Junsong Yuan. "Recognizing human actions as the evolution of pose estimation maps." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1159-1168. 2018.

[14] Martin, Manuel, Alina Roitberg, Monica Haurilet, Matthias Horne, Simon Reif3>, Michael Voit, and Rainer Stiefelhagen. "Drive&act: A multi-modal dataset for finegrained driver behavior recognition in autonomous vehicles." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2801-2810. 2019.

[15] Weyers, Patrick, David Schiebener, and Anton Kummert. "Action and object interaction recognition for driver activity classification." In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 4336-4341. IEEE, 2019.

Action recognition is one of the tasks in computer vision. There are many applications based on this, be it smart home or safety monitoring applications. In past years, there have been studies on this task such as I3D [8] and two-stream action recognition [9], that are based on RGB frames and optical flow. Recently human skeleton-based action recognition has made progress. Since ST-GCN [4] was proposed in 2018, more and more graph convolution networks (GCN) based models appear such as MS-AGCG [6], MS-G3D [3] and CTR-GCN [5], They all have a useful performance on the NTU-RGB+D dataset [2],

On the other hand, CNN-based models, for example Action Machine and PoseC3D, also make some achievements. In addition, for a better result, most models apply a fusion or ensemble method with different kinds of input data like RGB+skeleton.

In real world applications, object detection is a typical step before action recognition. Meanwhile, the goal is usually in a close relation to the action in both position and context aspects. For example, if a cup is detected and its position is close to an agent’s hand in a sequence of frames, this typically means that the bottle is moving along with the hand. Then there may be a high probability that the action is “drinking” which, however, may be confused with eating or a phone call. The known models have not made use of the object information. Some models are proposed for specific scenarios [10][14] and some choose body parts as objects rather than the object interacted with [15], In addition, the object information typically includes a non-intuitive distance matrix.

Image frames are a traditional input data format for action recognition. CNN is typically a good choice for solving image-based computer vision problems, including action recognition. Considering the temporal continuity of actions, a 3DCNN may be used as backbone model for extracting both spatial and temporal features, as disclosed in [11], for example. The input is typically a couple of continuous frames. Some research concentrates on multi-stream flows like I3D [8] and Two-Stream Network Fusion [9],

Skeleton-based action recognition is a recent topic in action recognition. The inputs are typically human body keypoints and/or the links, i.e. , the limbs, of them. This kind of data is more focused on human action itself and may reduce the impact of a complex background.

GCNs are a method to extract features from human skeletons. After ST-GCN [4] was proposed GCNs have made progress in action recognition tasks. Normally GCN- based models use the original coordinates and organize them as an undirected graph along with an adjacency matrix. MS-AAGCN [6] and MS-G3D [3] are two representative models. The former replaces fixed a human keypoints graph with an adaptive graph. The latter proposed an unbiased graph and an integrated spatial- temporal model.

Unlike GCNs that use coordinates directly, heatmaps illustrate keypoints or limbs on a blank image according to their position and optionally confidential score. Document [12] aggregates separate joint heatmaps into a motion one and then applies a 2DCNN model for classification. Some studies use encoded heatmaps rather than the pose one. For instance, in document [13], the pose heatmap is converted to an aggregated heatmap along spatial and temporal dimensions. PoseC3D is another known model. It uses separate joint heatmaps and comes up with a 3DCNN model which is able to achieve a high accuracy on the NTU RGB+D dataset. There are some works on exploring the object’s effect on action recognition. In document [10], a body-object descriptor is built by encoding the distance and angle between each object class and each joint. On the other hand, document [14] takes preset car-interiors as objects. It also uses the distance between objects and selected joints followed by a three-stream RNN network. Unlike the above two methods who have multiple objects, document [15] detects hands as objects together with 3D skeleton data. However, most models only apply to a specific environment, like a car-cabin, which is stable and actions are limited in.

US 11 367 313B2 discloses a method and apparatus for recognizing a body movement by sampling an input to-be-recognized video to obtain a sampled image frame sequence of the to-be-recognized video; performing keypoints detection on the sampled image frame sequence by using a trained body keypoints detection model, to obtain a body keypoint position heatmap of each sampled image frame in the sampled image frame sequence, the body keypoint position heatmap being used to represent a probability feature of a position of a preset body keypoint; and inputting body keypoint position heatmaps of the sampled image frame sequence into a trained movement classification model to perform classification, to obtain a body movement recognition result corresponding to the to-be-recognized video.

CN 106 022 213 B discloses a human motion recognition method based on three- dimensional bone information.

SUMMARY OF THE INVENTION

It is the object of the invention to improve methods for action recognition and vehicle occupant monitoring.

The object is achieved by the subject-matter of the independent claims. Preferred embodiments are subject-matter of the dependent claims.

The invention provides a computer-implemented method for recognizing an action performed by an agent in a sequence of frames, the method comprising: a) performing object detection on each frame to obtain object detection data that is indicative of an object position and an object class of an object present in the frames; b) generating a sequence of object class heatmaps for each object class that was recognized in the object detection data, wherein each object class heatmap includes information about at least one object of the respective object class; c) extracting from each frame key features of a skeleton of the agent and generating at least one sequence of key feature heatmaps; d) stacking each key feature heatmap for a specific timestep and each object class heatmap for the specific timestep obtained in steps b) and c) to obtain a sequence of combined heatmaps; e) feeding the sequence of combined heatmaps into a machine learning model that is configured as an action classifier and determining the action performed by the agent based on the sequence of combined heatmaps.

Preferably, in step a) the object position is determined as an object bounding box center of an object bounding box around the object.

Preferably, step c) comprises extracting from each frame limbs of the agent as key features and generating a limb heatmap as the key feature heatmap.

Preferably, step c) comprises extracting from each frame keypoints of the agent as key features and generating a keypoint heatmap as the key feature heatmap.

Preferably, in step b) each object class heatmap is generated by determining a distance between each recognized object of the same object class and all key features and including only that object into the object class heatmap that has the smallest distance.

Preferably, in step b) each object class heatmap is generated by determining a distance between each recognized object of the same object class and all keypoints and including only that object into the object class heatmap that has the smallest distance. Preferably, in step b) each object class heatmap is generated by including each recognized object of the same object class.

Preferably, in steps b) and c) each heatmap is generated with the same width and height as each frame on which the respective heatmap is based.

Preferably, in step d) the combined heatmap is generated with the same width and height as each previously generated heatmap, wherein the combined heatmap has a third dimension that corresponds to the sum of the amount of all key features and of all object classes that were previously determined.

Preferably, in step e) the machine learning model includes a 3DCNN that is trained as the action classifier.

The invention provides a computer-implemented method for monitoring an occupant of a vehicle and/or an agent outside the vehicle, the method comprising: a) recording the occupant and/or agent using an imaging device to obtain a sequence of frames; b) performing a previously described method on the sequence of frames of step a) to obtain an action performed by the occupant or agent; c) evaluating the action obtained in step b) and generating a control signal for the vehicle in accordance with the action.

Preferably, in step a) the object position is determined as a bounding box center of an object bounding box around the object.

Preferably, the control signal is configured to cause the vehicle to display a warning message, to initiate a braking procedure, and/or to perform an emergency braking procedure.

The invention provides a data processing device comprising means that are configured for performing a previously described method.

The invention provides a vehicle comprising an imaging device and a previously described data processing device. The invention provides a computer program that includes instructions that, upon execution on a data processing device, cause the device to perform a previously described method.

The invention provides a machine-readable data carrier or data carrier signal that comprises the computer program.

In this disclosure, a method that can be applied in more general action recognition tasks is proposed. Moreover, the disclosed method allows generation of a more intuitive data heatmap with class and position information.

In other words, a general method to generate heatmaps for potentially interacted objects is proposed. The idea includes both position and class information, and can be combined with skeleton information. In some embodiments a comparison of two approaches of the method can be made as a consistency check.

Human skeleton-based action recognition has been a popular topic in recent years. Many methods have made great advances such as MS-G3D (GCN-based) and PoseC3D (3DCNN-based). However, most models have not taken note to the role of objects. While those data usually can provide lots of useful information for the recognition network. In some embodiments a method is proposed to generate object data in heatmap format, that can be combined with human skeleton data to result in a better performance on action recognition. The disclosed method is verified on original and refined NTU-RGB+D 60 datasets. Compared with only using skeleton data, the combined data can improve the performance, specifically the accuracy by more than 1 percentage point.

In some embodiments object information is introduced into a human-skeleton based action recognition model. In some embodiments the framework detects objects frame by frame. In some embodiments 2D heatmaps for objects are generated and stacked together with keypoint or limb heatmaps. In some embodiments a 3DCNN based model such as PoseC3D is used to classify actions. This disclosure relates to computer vision, and more in particular, machine learning based algorithms for human action recognition that could be used in motor vehicle applications such as driver and occupant(s) status monitoring.

The present disclosure relates to computer vision technologies, and in particular, to a method for recognizing actions of agents, e.g., persons. In contrast to previous technologies, human actions are not only recognized based on human body features themselves, e.g., human body parts movement, but also based on objects that are interacted with. Human action is often associated with things/objects. For example, when people drink water, the hand and mouth are interacting with a water bottle, glass, mug or cup. In this disclosure, the object information is brought together with human body keypoints and/or skeletons to do action recognition so as to enable an improved recognition accuracy.

In some embodiments the objects are detected and located frame by frame of an input sequence of frames, i.e., an input video. In some embodiments a 2D heatmap containing object class information and location information is generated and stacked together with human body keypoint heatmaps, human bone heatmaps and/or limb heatmaps. In some embodiments the stacked heatmaps are sent to a trained 3DCNN model to recognize actions.

The method can be computer-implemented as software on an electronic device, and a storage medium of a desktop pc, laptop, or vehicle on-board computers, ECUs, and any other embedded computing platforms. This can be used in car cabin sensing products, including a driver monitoring system, a cabin monitoring system, as well as a car surround view, gesture-controlled infotainment systems, and the like.

In some embodiments each object class may be treated as one channel. In some embodiments the center of the object bounding box is drawn on the heatmap. In some embodiments, in each class channel, multiple same-class objects may appear. For example, two bottles may have been detected in action frames. There are two basic approaches. In the first approach, only one object for each class on that class heatmap is used. In each frame, for each same-class object, the distance between its center and each body keypoint may be determined. The minimum value is the minimum distance dmin of this object to the agent. So in each frame, in each class channel, the minimum distance of the m-th same-class object can be calculated as the minimum Euclidean the set of the body keypoints, x_m,y_m) is the center of the object bounding box, and (%fc,y_fe) is the coordinate of the fc-th keypoint. The object m whose dmin is smallest is chosen.

In the second approach all the objects that have the same class are included on that class heatmap. The method used to generate heatmap is Gaussian distribution, similar to document [1 ], So for the n-th object, its heatmap is defined as H_niJ = ■ .. . _r .. . . . , ,. , is the center of the object bounding box, s_n is a probability score and a is the variance of Gaussian distribution. The final heatmap size is K + N) x W x H, where K is the number of keypoints, N is the number of classes, and W and H are the width and height of the frame(s).

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention are described in more detail with reference to the accompanying schematic drawings.

Fig. 1 depicts an embodiment of a method for monitoring a vehicle occupant or agent outside the vehicle;

Fig. 2 depicts an embodiment of a method for recognizing an action of an agent;

Fig. 3 depicts a first approach for an object class heatmap; and

Fig. 4 depicts a second approach for an object class heatmap.

DETAILED DESCRIPTION OF EMBODIMENT Referring to Fig. 1 , an embodiment of a method for monitoring a vehicle occupant or an agent outside the vehicle includes a recording step S10. In the recording step S10 an imaging device, such as a camera, records a video. The imaging device may be installed so as to capture one or more vehicle occupants, preferably the driver, inside the vehicle cabin. It is also possible that the imaging device is installed to capture a portion of the surroundings of the vehicle and any agents, such as pedestrians, that are outside the vehicle. The video is organized as a sequence of individual frames in a manner known per se.

The method includes an action recognition step S12 in which the sequence of frames is analyzed. The action recognition step S12 determines an action that is performed by the vehicle occupant or agent by classification using a 3D convolution neural network. The action recognition step S12 may determine a semantic descriptor for the performed action.

The method includes a control step S14. In the control step S14, a control signal is generated in accordance with the previously determined action. The control signal may cause the vehicle to display a warning to the driver or initiate certain vehicle maneuvers such as braking, emergency braking, steering and the like.

Referring to Fig. 2 an embodiment of the action recognition step S12 is described in more detail. The action recognition step S12 comprises performing a method for recognizing an action performed by an agent 10, such as a vehicle occupant or an agent outside the vehicle, such as a pedestrian.

As previously described, initially a sequence of frames 12 of the agent 10 is recorded by an imaging device. The agent 10 is interacting with an object 14, such as a water bottle.

The sequence of frames 12 is fed to a skeleton recognition model that determines key features of a skeleton 16 of the agent 10 in each frame 12. A suitable skeleton recognition model can be chosen from ST-GCN (see document [4]), MS-AAGCN (see document [6]) or MS-G3D (see document [3]). The skeleton 16 includes a plurality of keypoints 18 and limbs 20. The keypoints 18 are also sometimes called joints.

The skeleton 16 is segmented into its constituents, i.e., the keypoints 18 and limbs 20. For each of the keypoints 18 a sequence of keypoint heatmaps 22 is generated. A specific keypoint heatmap 24 only includes one specific keypoint 26, e.g., the left hand, and excludes the other keypoints 28.

For each of the limbs 20 a sequence of limb heatmaps 30 is generated. A specific limb heatmap 32 only includes one specific limb 34, e.g., the left lower arm, and excludes the other limbs 36.

The keypoint heatmaps 22 and the limb heatmaps 30 are preferably generated using a Gaussian distribution model.

The sequence of frames 12 is also fed to an object detection model, e.g., FasterRCNN, that recognizes the object position and the object class of the object 14. The object position is preferably determined as the center of the object bounding box 38 around the recognized object 14, e.g., a water bottle.

For each object class a sequence of object class heatmaps 40 is generated.

Referring to Fig. 3, in a first approach, each object class heatmap 41 only includes one object 14, whereas another object 42 of the same class is excluded. The object class heatmap 41 is generated by determining the distance between each object 14 of the same object class and including only that object 14 that has the smallest distance to any of the keypoints 18, excluding any other objects 42.

Each keypoint heatmap, limb heatmap and object class heatmap of the same timestep in the sequence of keypoint heatmaps 22, the sequence of limb heatmaps 30 and the sequence of object class heatmaps 40 are combined into a combined heatmap 44. This results in a sequence of combined heatmaps 45, one combined heatmap 44 for each frame 46 of the sequence of frames 12. The sequence of combined heatmaps 45 is fed to a machine learning model 48. The machine learning model 48 includes a three-dimensional convolutional neural network. The machine learning model 48 is trained as an action classifier based on the combined heatmaps 44. The machine learning model 48 is preferably configured as a 3D convolutional neural network. A suitable 3D convolutional neural network is described in document [1], item 3.3 3D-CNN for Skeleton-based Action recognition in conjunction with Figure 2 there, the disclosure of which is incorporated herein by reference.

Consequently, the machine learning model 48 determines an action that was performed by the agent 10 based on information about the agent 10 itself, i.e. , the pose and movement key features such as keypoints 18 and limbs 20, and about the object 14 that is closest to the agent 10 in the sequence of frames 12. The action may be determined by a semantic descriptor or in the abstract, i.e., as some form of embedding. For example, the agent 10 may be recognized as drinking.

The action is fed to a control unit 50. The control unit 50 may be part of the vehicle and is configured to generate a control signal based on the previously recognized action performed by the agent 10. The control signal may cause the vehicle to maneuver, such as braking, emergency braking, steering and the like. It is also possible in the alternative or in addition to cause the vehicle to display a warning message to the driver.

Referring to Fig. 4, in the second approach, each object class heatmap 41 may include all objects 14 of the same object class.

The method was evaluated by the inventors using the dataset NTU RGB+D obtainable from (https://rose1.ntu.edu.sg/dataset/actionRecognition/). This is one of the most popular datasets for human skeleton-based action recognition. It includes 60 action classes with 40 subjects. In total 56,880 videos were collected by Kinect V2 with 3 different views and 17 setups. The dataset contains both 3D and 2D coordinates. In this evaluation only 2D RGB point data are used for combination with the object detection result. A refined version of the NTU RGB+D 60 from PoseC3D is used in the evaluation. In document [1], HRNet is used to regenerate the 2D coordinates of human skeletons, which can be more accurate than the original. Its skeleton has 17 joints while the original has 25 joints. The evaluation was conducted based on the NTU RGB+D 60 dataset that was processed by PoseC3D to reduce the number of joints.

Object Detection is done using FasterRCNN pretrained on the COCO dataset (available from https://cocodataset.org) as the detection model with 80 object classes.

A comparison of the disclosed model with the original PoseC3D (available at htps://github.com/open-mmlab/minaction2/tree/master/confiqs/skeleton/posec3d) shows an improvement of 1 .42 percentage points by using the single object per object class heatmap approach as shown in Table 1 . The model disclosed herein follows the same basic architechture, training parameters and training datasets. In addition to the PoseC3D model, the disclosed model includes a branch for detecting objects that are present and generating object class heatmaps that augment the conventional keypoint and limb heatmaps. With this augmentation an improvement in accuracy over the PoseC3D model without the object augmentation is achieved.

Table 1 Comparison of our method and original PoseC3D.

A comparison between the single object and multiple object approach, both according to the invention, shows that the single object approach obtains better results by 1 .83 percentage points, as shown in Table 2. Table 2 Comparison of single object and multiple object approach.

The disclosed method therefore is a valid alternative to the known action recognition methods and - depending on the approach - a significant improvement over the known methods.

REFERENCE SIGNS

10 agent

12 sequence of frames

14 object

16 skeleton

18 keypoint

20 limb

22 sequence of keypoint heatmaps

24 specific keypoint heatmap

26 specific keypoint

28 other keypoint

30 sequence of limb heatmaps

32 specific limb heatmap

34 specific limb

36 other limb

38 object bounding box

40 sequence of object class heatmaps

41 object class heatmap

42 another object

44 combined heatmap

45 sequence of combined heatmaps

46 frame

48 machine learning model

50 control unit

S10 recording step

S12 action recognition step

S14 control step

Claims

1 . A computer-implemented method for recognizing an action performed by an agent (10) in a sequence of frames (12), the method comprising: a) performing object detection on each frame (46) to obtain object detection data that is indicative of an object position and an object class of an object (14) present in the frames (46); b) generating a sequence of object class heatmaps (40) for each object class that was recognized in the object detection data, wherein each object class heatmap (41 ) includes information about at least one object (14) of the respective object class; c) extracting from each frame (46) key features (18, 20) of a skeleton (16) of the agent (10) and generating at least one sequence of key feature heatmaps (22, 30); d) stacking each key feature heatmap (22, 30) for a specific timestep and each object class heatmap (41 ) for the specific timestep obtained in steps b) and c) to obtain a sequence of combined heatmaps (45); e) feeding the sequence of combined heatmaps (45) into a machine learning model (48) that is configured as an action classifier and determining the action performed by the agent (10) based on the sequence of combined heatmaps (45).

2. The method according to claim 1 , wherein in step a) the object position is determined as an object bounding box center of an object bounding box (38) around the object (14).

3. The method according to any of the preceding claims, wherein step c) comprises extracting from each frame (46) limbs (20) of the agent (10) as key features and generating a limb heatmap as the key feature heatmap.

4. The method according to any of the preceding claims, wherein step c) comprises extracting from each frame (46) keypoints (18) of the agent (10) as key features and generating a keypoint heatmap as the key feature heatmap.

5. The method according to any of the preceding claims, wherein in step b) each object class heatmap (41 ) is generated by determining a distance between each recognized object (14) of the same object class and all key features and including only that object (14) into the object class heatmap (41 ) that has the smallest distance.

6. The method according to claims 4 and 5, wherein in step b) each object class heatmap (41 ) is generated by determining a distance between each recognized object (14) of the same object class and all keypoints (18) and including only that object into the object class heatmap (41 ) that has the smallest distance.

7. The method according to any of the preceding claims 1 to 4, wherein in step b) each object class heatmap (41 ) is generated by including each recognized object (14) of the same object class.

8. The method according to any of the preceding claims, wherein in steps b) and c) each heatmap (24, 32, 41 ) is generated with the same width and height as each frame (46) on which the respective heatmap (24, 32, 41 ) is based.

9. The method according to any of the preceding claims, wherein in step d) the combined heatmap (44) is generated with the same width and height as each previously generated heatmap (24, 32, 41 ), wherein the combined heatmap (44) has a third dimension that corresponds to the sum of the amount of all key features and of all object classes that were previously determined.

10. A computer-implemented method for monitoring an occupant of a vehicle and/or an agent (10) outside the vehicle, the method comprising: a) recording the occupant and/or agent (10) using an imaging device to obtain a sequence of frames (12); b) performing a method according to any of the preceding claims on the sequence of frames (12) of step a) to obtain an action performed by the occupant or agent (10); c) evaluating the action obtained in step b) and generating a control signal for the vehicle in accordance with the action.

11 . The method according to claim 10, wherein the control signal is configured to cause the vehicle to display a warning message, to initiate a braking procedure, and/or to perform an emergency braking procedure.

12. A data processing device comprising means that are configured for performing a method according to any of the preceding claims.

13. A vehicle comprising an imaging device and a data processing device according to claim 12.

14. A computer program that includes instructions that, upon execution on a data processing device, cause the device to perform a method according to any of the claims 1 to 11 .

15. A machine-readable data carrier or data carrier signal that comprises the computer program according to claim 14.