DE102021131930A1 - Gesture recognition in traffic - Google Patents

Abstract

According to a computer-implemented method for recognizing gestures in road traffic, a camera image (1) is obtained which depicts the surroundings of a motor vehicle (15). An area of interest of a person (2) directing road users is determined in the camera image (1). A super-resolution technique is applied to data dependent on the camera image (1) to generate an image representation of the area of interest of the driver (2) of the road user, the image representation having an increased resolution compared to the camera image (1). A gesture by the person (2) directing the road user is identified depending on the image representation.

Description

The invention is aimed at a computer-implemented method for recognizing gestures in road traffic, with a camera image being obtained that depicts the surroundings of a motor vehicle. The invention is also directed to a method for at least partially automatically driving a vehicle, to a computer-implemented training method for training an artificial neural network for gesture recognition in road traffic, to a data processing device, to an electronic vehicle guidance system and to a computer program product.

General pedestrian detection or person identification in busy scenes are known problems that are already addressed in the literature. However, in order to make autonomous or semi-autonomous vehicles safe and functional in certain situations, it is necessary to ensure that the deployed system is able to reliably direct people directing other road users using static or dynamic gestures, such as police officers or other law enforcement officials identify and recognize.

One object of the present invention is to achieve reliable gesture recognition in road traffic using a motor vehicle.

This object is achieved by the respective subject matter of the independent claims. Further designs and preferred embodiments are the subject of the dependent claims.

The invention is based on the idea, in a first step, to identify an area of interest of a person directing a road user in a camera image and then in a second step to apply an over-resolution technique in order to increase the resolution of this area of interest. A gesture of the person directing the road user can then be identified based on the area of interest with increased resolution.

According to one aspect of the invention, a computer-implemented method for gesture recognition in road traffic is provided. In this case, a camera image is obtained, in particular from a camera system of the motor vehicle or from a storage device of the motor vehicle, which images an environment of the motor vehicle. An area of interest of a person directing a road user in the camera image is determined. A super-resolution technique is applied to data dependent on the camera image, specifically the camera image and the area of interest of the road user driver, to generate an image representation of the area of interest of the road user driver, the image representation having an increased resolution relative to the camera image has. A gesture by the person directing the road user is identified depending on the image representation.

In particular, the super-resolution technique is only applied to the portion of the camera image that corresponds to the area of interest and is not applied to other areas of the camera image outside the area of interest.

Unless stated otherwise, all steps of the computer-implemented method can be carried out by at least one computing unit. In particular, a data processing device with at least one processor that is set up or adapted to carry out a computer-implemented method according to the invention can carry out the steps of the computer-implemented method. For this purpose, the data processing device, which can correspond to the at least one computing unit, can in particular store a computer program which contains instructions which, when executed by the data processing device, in particular the at least one processor, cause the data processing device to execute the computer-implemented method .

The person directing the road user may be a person who directs road users, including for example the motor vehicle and/or other motor vehicles, in traffic using static and/or dynamic gestures. The person directing the road user can be recognized as such, for example, on the basis of characteristic clothing, such as a characteristic uniform, safety clothing, hats, etc. Alternatively or additionally, the person steering the road user can be recognized as such based on characteristic poses or body orientations or behavior in road traffic. For example, the road user driver may be a police officer or other law enforcement officer who is visually identifiable as such.

An object detection algorithm can be applied to determine the area of interest. The object recognition algorithm can be executed, for example, by a trained artificial neural network or a corresponding detection module of such an artificial neural network.

An object recognition algorithm can be understood as a computer algorithm that is able to recognize one or more objects within a provided input image by specifying respective areas of interest, also referred to as bounding boxes, and assigning a respective object class to each of the bounding boxes, where the object classes can be selected from a predetermined set of object classes. In this context, the assignment of an area of interest can be understood in such a way that a corresponding confidence value or a corresponding probability is provided for the object identified within the area of interest belonging to the corresponding object class. For example, the algorithm can provide such a confidence value or probability for each of the object classes for a given area of interest. The assignment of the object class may involve, for example, selecting or providing the object class with the highest confidence value or the highest probability.

Regarding the object detection algorithm that may be used to determine the area of interest for the road user driver, one of the object classes may be a road user driver class. However, it is emphasized that determining the area of interest of the driver does not necessarily take place in one step or using a single object detection algorithm. For example, in some implementations, an object detection algorithm is applied to identify pedestrians or people in the camera image that correspond to certain classes, such as people standing or walking, etc. These pedestrians or people can include the person directing the road user, who can then be selected or identified in a second step.

In particular, to determine the area of interest of the person directing road users, a position and/or a shape of the area of interest in the camera image can be determined and it can be determined that the respective object class is "person directing road users".

A gesture of the person directing the road user can be a static or a dynamic gesture. Typically, gestures of the person directing the road user involve characteristic positions and/or orientations of the person's body or body part, face or head or hands, et cetera. Gestures by the driver may also include characteristic positions and/or orientations of an implement, such as a trowel, held by the driver. Such characteristic positions and/or orientations can be kept essentially constant for a certain period of time, which can cover several camera frames, with the camera image corresponding to one of the camera frames. After that, the person driving the road user can change his pose to signal different information to the road users. In the context of the present disclosure, a pose during a period of time during which the pose of the driver is substantially constant may also be considered a gesture. In other words, a gesture does not necessarily involve actual movement of the body or implement.

The image representation of the area of interest can be understood as an image that depicts the part of the camera image that corresponds to the area of interest, but with an increased resolution compared to the corresponding area in the camera image. A number of algorithms are known per se that achieve an over-resolution of an image. A super-resolution technique can be viewed as a process for obtaining high-resolution images from lower-resolution images. There are well-known super-resolution models based on deep learning as well as classic upsampling techniques such as interpolation-based methods that can be used as super-resolution techniques.

Known algorithms, for example algorithms based on deep learning, can also be used to recognize the gesture based on the image representation.

According to a computer-implemented method according to the invention, a two-step approach is used to achieve reliable gesture recognition in road traffic environments. In particular, in the first step the area of interest of the person directing the road user is determined. Once this is done, the super-resolution technique is applied specifically to the region of interest to obtain the enhanced resolution image representation. By determining the gesture based on the image representation with increased resolution, a particularly reliable gesture recognition is possible, since it takes advantage of the increased resolution without being disturbed by additional information outside the area of interest, which is not useful for recognizing the gesture. Furthermore, by confining the super-resolution technique to the area of interest, computational effort and memory requirements are reduced.

The recognized gesture can then be used by an electronic vehicle guidance system of the motor vehicle for autonomous or semi-autonomous driving of the motor vehicle depending on the identified gesture. For example, the electronic vehicle guidance system may execute a trajectory planning algorithm or the like based on the gesture. Since the gesture is determined according to the invention with improved reliability, the safety and reliability as well as the availability of autonomous and semi-autonomous driving are increased.

According to some embodiments of the gesture recognition method according to the invention, a detection module of a trained artificial neural network is applied to the camera image in order to determine the area of interest of the person driving the road user in the camera image.

Implementations utilizing the artificial neural network allow for quick and reliable detection of the area of interest and/or quick and reliable generation of the enhanced resolution image representation.

According to some implementations, a super-resolution module of the artificial neural network is applied to the data dependent on the camera image to apply the super-resolution technique.

The super-resolution module is a trained module of the trained artificial neural network. This also applies to all other modules and sub-modules of the trained artificial neural network, which can be introduced below for specific implementations, unless stated otherwise.

According to some implementations, the super-resolution module includes a transposed convolution layer and/or sub-pixel layer.

In particular, the transposed convolution layer and/or the sub-pixel layer is applied to the data that depends on the camera image, in particular to the part of the camera image that corresponds to the area of interest, to generate the image representation.

In this way, unwanted side effects of other approaches to over-resolution, especially upsampling techniques, including computational complexity, noise amplification, blurring, et cetera, can be reduced.

According to some implementations, a pedestrian detection sub-module of the detection module is applied to the camera image to generate intermediate data describing a respective area of interest for each of the one or more pedestrians in the image. A classification sub-module of the detection module is applied to the intermediate data to classify one of the one or more pedestrians as a driver of the road user.

In other words, determining the area of interest of the person directing the road user is carried out in two steps by the respective sub-modules of the artificial neural network. In the first step, an object detection algorithm can be applied to identify all pedestrians in the camera image with their respective area of interest and optionally corresponding pedestrian classes. The different pedestrian classes can include, for example, classes such as standing pedestrians, walking pedestrians, or seated pedestrians, et cetera. However, the output of the pedestrian detection sub-module, in particular the intermediate data, does not contain any information as to whether the individual area of interest corresponds to a person driving the road user or not. In order to classify the person driving the road user, or in other words to select or determine the correct area of interest from the different areas of interest for the pedestrians obtained in the first step, the intermediate data is forwarded to the classification sub-module.

For example, the pedestrian detection sub-module may include an encoder module for encoding features of the camera image. The pedestrian detection sub-module can then determine the areas of interest for the pedestrians and the pedestrian classes based on the encoded features. The super-resolution module may act or comprise a corresponding decoder module that reconstructs the image representation from the encoded features.

According to several implementations, a current pose of the person driving the road user is determined based on the image representation, for example by applying a pose estimation module of the artificial neural network to the image representation. The gesture of the person directing the road user is identified depending on the current pose.

The current pose of the person driving the road user can be regarded as a body pose of the person driving the road user as shown in the camera image. The body pose may include, for example, a position and/or orientation of the body and/or face, arms, hands, head, et cetera, of the person driving the road user. Furthermore, the position and/or orientation of an implement, e.g. a traffic signal indicator et cetera, may be considered part of the body pose, even though the implement is not a part of the body of the person driving the road user.

For example, to determine the current pose of the driver, certain predetermined key points of the driver may be determined, for example by the pose estimation module. The key points can refer to characteristic points, for example in the middle of the hands or the face or the trowel. Based on the position of the key points, the gesture can be learned or determined and it can also be determined whether the gesture corresponds to a signal directed to the motor vehicle or its user himself or to another vehicle in the vicinity. For example, prior art human pose estimation networks can be used to learn and identify different poses and gestures appropriately.

In particular, a gesture recognition module of the artificial neural network can be applied to input data containing the current pose to identify the gesture of the driver.

According to some implementations, the gesture recognition module includes a gesture validation sub-module and a recurrent neural network, RNN, which may be configured, for example, to include or consist of a long short-term memory, LSTM. The gesture validation sub-module is applied to the input data and the RNN is applied to an output of the gesture validation sub-module. An output of the RNN is fed back to the gesture validation submodule as part of the input data. The gesture is identified based on the output of the RNN.

In particular, the described steps of the computer-implemented method can be performed for a plurality of camera images corresponding to respective consecutive camera frames. Due to the described construction of the gesture recognition module with the RNN, which feeds back its output to the gesture validation sub-module, several consecutive frames can be taken into account in order to achieve a particularly reliable recognition of the correct gesture.

In some cases, the meaning of the current pose is not clear either, but may depend on previous poses of the person driving the road user. By considering the RNN output as described, such gestures can also be reliably identified.

According to a further aspect of the invention, a computer-implemented training method for training an artificial neural network for gesture recognition in road traffic is provided. A training image is obtained that depicts the surroundings of a motor vehicle. A trained artificial neural network detection module is applied to a training data set dependent on the training image to determine an area of interest of a driver in the training image. A super-resolution module of the artificial neural network is applied to data dependent on the training data set to generate an image representation of the area of interest of the driver, the image representation having increased resolution over the training image. A gesture by the person directing the road user is identified depending on the image representation. A loss function, in particular a common loss function, for training the artificial neural network is evaluated depending on the gesture.

In particular, the described steps of the training method can be viewed as part of a single training epoch of training. The steps can be repeated several times for several training epochs until a predetermined termination criterion is met and the artificial neural network can be considered sufficiently trained.

In particular, the steps can also be repeated for a sequence of training images for each of the training epochs, in particular if the gesture recognition module of the artificial neural network has the RNN.

In particular, the artificial neural network can be trained in an end-to-end manner based on the joint loss function.

According to some embodiments of the inventive computer-implemented training method, a masking and prediction module is applied to the training image to generate the training data set, the training data set corresponding to a modified version of the training image, wherein the area of interest of the person driving the road user is partially obscured, in particular is artificially covered.

For example, the occlusion may be performed in a random manner with a certain probability and/or shape and/or to a certain extent according to a given level of difficulty. The higher the probability of occlusion, or the larger the occluded portion of the area of interest, the greater the difficulty.

In this way, the robustness of the trained artificial neural network can be increased since not only the original training images but also at least partially covered versions of the training image are used for the training.

According to some implementations, the difficulty level is determined according to a curriculum learning strategy, and an occlusion level and/or a occlusion probability of the area of interest of the road user driver depends on or is determined dependent on the difficulty level.

Curriculum learning can be viewed as a training strategy that aims to gradually increase the complexity or difficulty of the data fed to the artificial neural network as training input. In this way, simpler objects can be trained first and the object difficulty can be gradually increased. In combination with the masking-and-predicting strategy, the person driving the road user can be increasingly masked.

According to some embodiments of the computer-implemented method for traffic gesture recognition according to the invention, wherein the detection module is used to determine the area of interest and/or the super-resolution module is used to apply the super-resolution technique, the artificial neural network is or has been generated using an inventive computer-implemented training method.

According to a further aspect of the invention, a method for at least partially automatically driving a motor vehicle is provided. In this case, a multiplicity of camera images, which correspond to successive camera frames, are generated using a camera of the vehicle. For each of the camera images, a computer-implemented method according to the invention for gesture recognition is executed by a computing unit of the motor vehicle. The motor vehicle is guided at least partially automatically depending on the specific gesture.

In particular, the at least one computing unit can identify a traffic rule or an instruction encoded by the identified gesture, and the motor vehicle is guided at least partially automatically depending on the traffic rule or the instruction.

In order to guide the motor vehicle at least partially automatically, the computing unit or another computing unit of the motor vehicle or a control unit of the motor vehicle can, for example, generate respective control signals for one or more actuators of the motor vehicle, which enable longitudinal control and/or lateral control of the motor vehicle in an automated or partially automated manner implement or influence based on the control signals.

According to a further aspect of the invention, a data processing device with at least one processor is provided, which is set up to execute a computer-implemented method according to the invention or a computer-implemented training method.

For example, the computing unit of the motor vehicle can correspond to such a data processing device that is set up to execute a computer-implemented method according to the invention.

According to a further aspect of the invention, an electronic vehicle guidance system is provided for at least partially automatically guiding a motor vehicle depending on a gesture by a person steering the road user in the vicinity of the motor vehicle. The electronic vehicle guidance system has a camera that is set up to generate a large number of camera images that correspond to successive camera images, and a processing unit that is set up to determine the gesture by executing a computer-implemented method for gesture recognition according to the invention for each of the camera images .

Further versions of the electronic vehicle guidance system result directly from the various versions of the method according to the invention for at least partially automatically driving a motor vehicle and vice versa. In particular, an electronic vehicle guidance system according to the invention can be set up or programmed to carry out a method according to the invention, or carries out such a method.

According to some implementations, a computer program product including instructions is provided. If the commands from a data processing device, the instructions cause the data processing device to perform a computer-implemented method or a computer-implemented training method according to the invention.

According to a further aspect of the invention, a computer-readable storage medium is provided which stores a computer program according to the invention.

The computer program and the computer-readable storage medium can be referred to as a computer program product containing respective instructions.

An electronic vehicle guidance system can be understood to mean an electronic system that is set up to guide a vehicle fully automatically or fully autonomously, in particular without manual intervention or manual control by a driver or user of the vehicle being required. The vehicle automatically carries out all necessary functions such as steering, braking and/or acceleration manoeuvres, as well as monitoring and registering road traffic and responding accordingly. In particular, the electronic vehicle guidance system can implement a fully automatic or fully autonomous driving mode according to level 5 of the classification according to SAE J3016. An electronic vehicle guidance system can also be implemented as a driver assistance system (English: "advanced driver assistance system", ADAS), which supports the driver in partially automated or partially autonomous driving. In particular, the electronic vehicle guidance system can implement a partially automated or partially autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, "SAE J3016" refers to the corresponding standard in the June 2018 version.

The at least partially automatic vehicle guidance can therefore include driving the vehicle according to a fully automatic or fully autonomous driving mode according to level 5 of the classification according to SAE J3016. The at least partially automatic vehicle guidance may also include driving the vehicle according to a partially automated or partially autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

A module or sub-module may be understood to mean a piece of software code that is operably linked and combined into a unit and is stored in a computer-readable manner on a computer-readable storage medium. A module or sub-module may have or implement multiple processing steps and/or data structures.

A computing unit can be understood in particular as a data processing device which has a processing circuit. The arithmetic unit can therefore, in particular, process data for carrying out arithmetic operations. This also includes operations to perform indexed accesses to a data structure, for example a look-up table (LUT).

In particular, the processing unit can contain one or more computers, one or more microcontrollers and/or one or more integrated circuits, for example one or more application-specific integrated circuits, ASICs (English: “application-specific integrated circuit”), one or more field-programmable gate arrays , FPGA, and/or one or more single-chip systems, SoC (English: "system on a chip"). The computing unit can also have one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processor units, GPU and/or contain one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit can also contain a physical or a virtual network of computers or other of the units mentioned.

In various embodiments, the computing unit contains one or more hardware and/or software interfaces and/or one or more memory units.

A memory device can be configured as volatile data storage, such as dynamic random access memory (DRAM), or static random access memory (SRAM), or non-volatile Data memory, for example as a read-only memory, ROM, as a programmable read-only memory, PROM, as an erasable read-only memory, EPROM (erasable read-only memory) ), as electrically erasable read-only memory, EEPROM (English: "electrically erasable read-only memory"), as flash memory or flash EEPROM, as ferroelectric memory with random access, FRAM (English: "ferroelectric random access memory"), as magnetoresistive random access memory, MRAM or phase change random access memory Access, PCRAM (English: "phase-change random access memory"), be configured.

Further features of the invention result from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description and the features and combinations of features mentioned below in the description of the figures and/or shown in the figures can be included in the invention not only in the combination specified in each case, but also in other combinations. In particular, embodiments and combinations of features that do not have all the features of an originally formulated claim are also covered by the invention. In addition, embodiments and combinations of features that go beyond or deviate from the combinations of features set out in the back references of the claims are encompassed by the invention.

• 1 schematisch eine Straßennutzer lenkende Person;
• 2 schematisch eine weitere Straßennutzer lenkende Person;
• 3 schematisch ein Kraftfahrzeug mit einer beispielhaften Ausführung eines erfindungsgemäßen elektronischen Fahrzeugführungssystems;
• 4 schematisch ein trainiertes künstliches neuronales Netzwerk zur Verwendung in einer beispielhaften Ausführung des erfindungsgemäßen computerimplementierten Verfahrens zur Gestenerkennung;
• 5 schematisch das Wirkungsprinzip eines Gestenerkennungsmoduls eines künstlichen neuronalen Netzwerks zur Verwendung in einer weiteren beispielhaften Ausführung eines erfindungsgemäßen computerimplementierten Verfahrens zur Gestenerkennung; und
• 6 schematisch eine Architektur eines künstlichen neuronalen Netzwerks, das gemäß einer beispielhaften Ausführung eines erfindungsgemäßen computerimplementieren Trainingsverfahrens trainiert werden soll.

Show in the figures

• 1 schematic of a person steering a road user;
• 2 schematically another person directing road users;
• 3 schematically a motor vehicle with an exemplary embodiment of an electronic vehicle guidance system according to the invention;
• 4 schematically shows a trained artificial neural network for use in an exemplary embodiment of the computer-implemented method for gesture recognition according to the invention;
• 5 schematically shows the operating principle of a gesture recognition module of an artificial neural network for use in a further exemplary embodiment of a computer-implemented method for gesture recognition according to the invention; and
• 6 schematically shows an architecture of an artificial neural network that is to be trained according to an exemplary embodiment of a computer-implemented training method according to the invention.

General pedestrian detection or person identification in busy scenes are common problems and are already addressed in the literature. However, in order to make autonomous vehicles workable in some areas, manufacturers need to ensure that the installed system is capable of showing road users driving persons 2, such as law enforcement officers, e.g. police officers, in camera images 1, as in 1 and in 2 shown schematically, to identify and recognize in most or all cases including animated scenes, and to follow the instructions accordingly by interpreting their static or dynamic gestures.

3 shows schematically a motor vehicle 15 with an exemplary embodiment of an electronic vehicle guidance system 18 according to the invention. The electronic vehicle guidance system 18 has a camera 16, for example a front camera of the motor vehicle 15, which is set up to generate a large number of camera images 1, the successive camera Frames correspond, and a computing unit 17, for example one or more electronic control units, ECUs (English: electronic control unit), of the motor vehicle 15. The computing unit 17 is set up to the gesture of a road user directing person 2 in the camera images 1 by executing a to determine computer-implemented method according to the invention for gesture recognition in road traffic for each of the camera images 1. In other words, the electronic vehicle guidance system 18 can execute a method for at least partially automatically guiding the motor vehicle 15 depending on the gesture of the person 2 steering the road user.

For this purpose, a memory device (not shown) of the electronic vehicle guidance system 18, for example the computing unit 17, stores a trained artificial neural network, ANN (English: artificial neural network), 3, which is shown schematically in 4 is shown and may be trained, for example, using a computer-implemented method according to the invention.

The ANN 3 comprises, for example, a detection module 5 which may be part of a key point detection model 4 in some implementations. The detection module 5 receives the camera image 1 of a given camera frame as an input and determines a region of interest, ROI (region of interest), of the person directing the road user 2 in the camera image 1. The ANN 3 further has a super-resolution module 6 on, for example as part of the key point detection model 4. The super-resolution module 6 receives output data from the detection module 5 and applies a super-resolution technique to generate an image representation of the ROI of the road user directing person 2, the image representation having an improved resolution compared to the original camera image 1 .

Image superresolution is a process of restoring high resolution images from low resolution images. With cont As deep learning techniques have advanced, deep learning-based super-resolution models have been actively explored and often perform very well on various super-resolution benchmark tests. Various deep learning methods have been applied to solve super-resolution problems, ranging from methods based on convolutional neural networks (CNN) to approaches based on generative adverse networks (GAN) are sufficient. To apply super-resolution techniques, the input can be upsampled. In general, there are two approaches to applying upsampling: oversampling before upsampling and oversampling after upsampling. There are several classic upsampling techniques including, for example, interpolation-based methods. To overcome the disadvantages of interpolation-based methods, such as complexity, noise amplification, fuzzy results et cetera, and to learn upsampling in an end-to-end manner, transposed convolutional layers and sub-pixel layers can be used.

The ANN 3 may also include a pose estimation module 11, for example as part of the key point detection model 4. The pose estimation module 11 may receive the enhanced resolution image representation as input and determine a current pose of the road user driver 2 as output.

For example, to identify the gesture signals of the driver 2, the pose estimation module 11 may identify characteristic key points, such as the hands and face of the driver 2. Based on the positions of the key points, the model may have learned to identify the gesture made by the person 2 driving the road user and also to determine whether the signal given is intended for the motor vehicle 15 or for another road user. In the key point recognition model 4, prior art human pose estimation networks can be used to learn and identify different poses and learn the gestures accordingly.

The ANN 3 may also comprise a gesture recognition module, for example in addition to the key point recognition model 4, which may contain a gesture validation sub-module 12 and a long short-term memory, LSTM, 13. The gesture validation sub-module 12 is applied to input data containing the output of the pose estimation module 11, and the LSTM 13 receives a respective output of the gesture validation sub-module 12. An output of the LSTM 13 is fed back to the gesture validation sub-module 12 as part of the input data.

The gesture made by the person 2 directing the road user can be of different types with different meanings. The traffic control gesture signal may also vary from country to country. Some gesture signals can be performed with moving hands and with different movements and therefore cannot necessarily be reliably determined within a single camera frame. Thus, recognizing these types of gestures may leverage input from previous camera frames. The gesture validation sub-module 12 thus has the feedback link from the previous camera frames to determine and learn all types of gestures. These feedback connections come from the LSTM 13 which is responsible for learning the previous frame dependencies.

LSTM networks are a type of recurrent neural networks, RNN, capable of learning order dependencies in sequence prediction problems. RNN differ from traditional feedforward neural networks. Rather than just feeding in input data, it also takes previous historical information to predict the next outcome in the sequence. The LSTM 13 is able to learn long-term dependencies.

In 5 the working principle of the LSTM 13 is explained. For different camera frames or timestamps ti, with i=1 ..3 for example, one has the feedback of the previous timestamps t _i-1 ., coming from the LSTM 13, to the gesture validation sub-module 12. The gesture validation sub-module 12 can base the signals based on the information of the current frame and the information of the previous frames.

The ANN 3 can be trained in an end-to-end manner, for example using a computer-implemented training method according to the invention, as schematically illustrated in FIG 6 shown.

For example, a curriculum learning strategy can be used to train the ANN3. Curriculum learning is a training method that aims to gradually increase the complexity of the data fed to the ANN 3 input. The central idea is to train using simple objects first and gradually increase the difficulty of the objects. In In most real-world scenarios, the objects are at least partially obscured by other objects, which can lead to incorrect predictions. To address this problem, training can employ a masking-and-prediction strategy. Masking-and-prediction is a particular strategy that can be used in connection with curriculum learning, for example. Here, the person 2 driving the road user can be increasingly masked by means of a masking and prediction module 14, and the ANN is intended to predict bounding boxes of visible areas and masked areas.

The masked image can be passed to an encoder sub-module 9 of the pedestrian detection sub-module 7 for feature extraction. Known deep neural network encoder architectures, for example ResNet or VGG, can be used. The extracted features are then passed to multiple convolution layers 10 which are responsible for detecting all pedestrians in the camera 16 field of view. A classification sub-module 8 of the detection module 5 can classify the detected pedestrians and filter out the ROI of the person 2 directing road users in the scene.

The ROI of the person 2 driving the road user is then forwarded to the super-resolution module 6, which is responsible for increasing the quality and resolution of the ROI of the person 2 driving the road user. The improved ROI is then forwarded to the pose estimation module 11 which is responsible for recognizing the key points and for estimating the pose of the driver 2 based on these key points, for example the relative positions of the hands and the face orientation. The pose is used to recognize the type of gestures that the driver 2 is trying to make. After the pose estimation, the extracted information is passed to the gesture validation sub-module 12 . This module finally determines the type of gestures made by the person 2 directing the road user. Some of the gestures require moving the hands or performing some other type of movement. Recognizing this type of gesture may require the information from previous frames, which is achieved using the LSTM 13, which takes into account the historical information about the scene.

The explanations regarding the computer-implemented training method according to the invention can be transferred analogously to the computer-implemented method for gesture recognition according to the invention, with the exception that the masking and prediction module 14 is no longer required after the training has been completed.

Claims (15)

Computer-implemented method for recognizing gestures in road traffic, a camera image (1) being obtained which depicts the surroundings of a motor vehicle (15), characterized in that - an area of interest of a person (2) steering the road user is determined in the camera image (1). ; - applying a super-resolution technique to data dependent on the camera image (1) to generate an image representation of the area of interest of the person driving the road user (2), the image representation having an increased resolution compared to the camera image (1); and - a gesture by the person (2) directing the road user is identified as a function of the image representation. Computer-implemented method claim 1 , characterized in that - a detection module (5) of a trained artificial neural network (3) is applied to the camera image (1) in order to determine the area of interest of the person (2) directing the road user in the camera image (1); and/or - a super-resolution module (6) of the artificial neural network (3) is applied to the data dependent on the camera image (1) in order to apply the super-resolution technique. Computer-implemented method claim 2 , characterized in that - a pedestrian detection sub-module (7) of the detection module (5) is applied to the camera image (1) to generate intermediate data describing a respective area of interest for each of one or more pedestrians in the camera image (1). ; and - a classification sub-module (8) of the detection module (5) is applied to the intermediate data to classify one of the one or more pedestrians as the person (2) driving the road user. Computer-implemented method according to one of claims 2 or 3 , characterized in that - a current pose of the person (2) driving the road user is determined based on the image representation, in particular using a pose estimation module (11) of the artificial neural network (3) on the image representation; and - the gesture of the person (2) driving the road user is identified depending on the current pose. Computer-implemented method claim 4 , characterized in that a gesture recognition module (12, 13) of the artificial neural network (3) is applied to input data containing the current pose to identify the gesture of the person (2) driving the road user. Computer-implemented method claim 5 , characterized in that - the gesture recognition module (12, 13) comprises a gesture validation sub-module (12) and a recurrent neural network (13); - the gesture validation sub-module (12) is applied to the input data and the recurrent neural network (13) is applied to an output of the gesture validation sub-module (12); - an output of the recurrent neural network (13) is fed back to the gesture validation sub-module (12) as part of the input data; and - the gesture is identified based on the output of the recurrent neural network (13). Computer-implemented method according to one of claims 2 until 6 , characterized in that the super-resolution module (6) has a transposed convolution layer and/or a sub-pixel layer. Computer-implemented training method for training an artificial neural network (3) for gesture recognition in road traffic, a training image being obtained which depicts the surroundings of a motor vehicle (15), characterized in that - a detection module (5) of a trained artificial neural network (3) is applied to a training data set dependent on the training image to determine an area of interest of a person driving the road user (2) in the training image; - a super-resolution module (6) of the artificial neural network (3) is applied to data dependent on the training data set in order to generate an image representation of the area of interest of the person (2) directing the road user, the image representation having an increased resolution compared to the training image has; - a gesture by the person (2) directing the road user is identified as a function of the image representation; - a loss function for training the artificial neural network (3) is evaluated depending on the gesture. Computer-implemented training method claim 8 , characterized in that a masking and prediction module (14) is applied to the training image to generate the training data set, the training data set corresponding to a modified version of the training image, the area of interest of the person directing the road user (2) is partially covered. Computer-implemented training method according to one of the Claims 8 or 9 , characterized in that - a degree of difficulty is determined according to a curriculum learning strategy; and - a degree of concealment and/or a probability of concealment of the area of interest of the person (2) directing the road user depends on the degree of difficulty. Computer-implemented method according to one of claims 2 until 7 , characterized in that the artificial neural network (3) using a computer-implemented training method according to one of Claims 8 until 10 is trained. A method for at least partially automatically driving a motor vehicle (15), wherein - a multiplicity of camera images (1), which correspond to successive camera frames, are generated using a camera (16) of the motor vehicle (15); - For each of the camera images (1), a computer-implemented method according to one of Claims 1 until 7 or 11 is executed by a computing unit (17) of the motor vehicle (15); and - the motor vehicle (15) is guided at least partially automatically as a function of the gesture. Data processing device having at least one processor which is set up to execute a computer-implemented method according to one of Claims 1 until 7 or 11 execute or a computer-implemented training method according to one of Claims 8 until 10 to execute. Electronic vehicle guidance system (18) for at least partially automatically driving a motor vehicle (15) depending on a gesture by a person (2) steering a road user in the vicinity of a motor vehicle (15), wherein the electronic vehicle guidance system (18) - has a camera (16), arranged to generate a plurality of camera images (1) corresponding to consecutive camera frames; and - a computing unit (17) which is set up to, by executing a computer-implemented method according to one of Claims 1 until 7 or 11 to determine the gesture for each of the camera images (1). Computer program product, which contains instructions which, when executed by a data processing device, cause the data processing device to carry out a computer-implemented method according to one of Claims 1 until 7 or 11 execute or a computer-implemented training method according to one of Claims 8 until 10 to perform.

Images