DE102018206777B4 - Depth evaluation for the localization of a vehicle occupant in a vehicle interior - Google Patents

Abstract

Depth evaluation device (10) for a vehicle (1) for localizing a vehicle occupant (2) in a vehicle interior (3) comprising • a first input interface (11) to receive first image data of a recording of the vehicle occupant (2), • a second input interface ( 12) in order to obtain second image data of a recording of the vehicle occupant (2), • the first image data being the image data of a mono camera (4) and the second image data being the image data of a 3D camera (5), • an artificial neural network (14 ), o wherein the depth evaluation device (10) is designed to execute the artificial neural network (14) when the artificial neural network (14) is loaded in a memory (16) of the depth evaluation device (10), o wherein the artificial neural network ( 14) is designed to obtain first depth information from the first image data as a function of the first image data and, as a function of the second image data, to obtain second depth information from the to obtain second image data, • the depth evaluation device (10) being designed, o to obtain a first surface model of the vehicle occupant (2) as a function of the first depth information, o to obtain a second surface model of the vehicle occupant (2) as a function of the second depth information, o as a function of of the first surface model to calculate the localization of the vehicle occupant (2) in the vehicle interior (3), o to calculate the localization of the vehicle occupant (2) in the vehicle interior (3) as a function of the second surface model, o wherein the depth evaluation device (10) has a logic module (18), which is designed to merge the first surface model with the second surface model, o wherein the depth evaluation device (10) is designed to calculate the localization of the vehicle occupant (2) in the vehicle interior (3) as a function of this fusion and an output interface (17) to at least the locator provision of the vehicle occupant (2).

Description

The invention relates to a depth evaluation system for a vehicle for locating a vehicle occupant in a vehicle interior according to claim 1. Furthermore, the invention relates to a depth evaluation system for a vehicle for locating a vehicle occupant in a vehicle interior according to claim 4. The invention also relates to a training method for an artificial neural network for calculating depth information according to claim 9. Furthermore, the invention relates to a method for localizing a vehicle occupant in a vehicle interior according to claim 10. The invention also relates to a computer program product for localizing a vehicle occupant in a vehicle interior according to claim 12.

A three-dimensional structure of the surroundings can be calculated from image data from a mono camera with the aid of the optical flow when the camera moves relative to the surroundings. Prior art is disclosed in, for example DE 10 2011 113 081 A1 . However, this is not the case in the interior of a vehicle, since the camera and occupants do not move towards one another.

Furthermore, methods are known for directly estimating 2- or 3-dimensional skeletal models of people from mono camera images. Prior art is disclosed, for example, in Dushyant Mehta et al., “Monocular 3D Human Pose Estimation Using Transfer Learning and Improved CNN Supervision”, http://arxiv.org/abs/1611.09813. However, these do not offer the full surface description of the person and / or require some form of movement of the camera relative to the objects.

1. (1) Erkennung der Abgelenktheit des Fahrers, um abschätzen zu können, welche Zeit der Fahrer benötigt, um eine potentielle Nebentätigkeit zu beenden und sich wieder komplett der Fahraufgabe zu widmen;
2. (2) Erkennung der Müdigkeit des Fahrers;
3. (3) exakte Lokalisierung der Insassen, um bei einer möglichen Kollision oder einem Verkehrsunfall Fahrzeuginsassen-Rückhaltesystems, wie zum Beispiel Gurte, Innenairbags und/oder Außenairbags, optimal parametrieren zu können.

In order to achieve a higher degree of automation when driving a vehicle, interior cameras are used that have several functions, including:

1. (1) Detection of the distraction of the driver in order to be able to estimate how long the driver needs to end a potential secondary activity and to devote himself completely to the driving task again;
2. (2) detection of driver fatigue;
3. (3) Exact localization of the occupants in order to be able to optimally parameterize vehicle occupant restraint systems such as belts, interior airbags and / or exterior airbags in the event of a possible collision or a traffic accident.

Mono cameras are often used in the vehicle interior. However, no three-dimensional surfaces and / or models of the body can currently be calculated with mono cameras in the vehicle interior. Therefore, (1) and (2) can only be determined with limited precision. The exact localization of the occupants according to (3) is only possible very imprecisely. For this reason, sensors are also used here that provide depth information, such as cameras that measure distances using runtime methods. Such cameras are called Time-Of-Flight, abbreviated TOF, cameras. Stereo cameras also provide depth information. With such cameras, (1) - (3) can be satisfactorily covered. However, such cameras are significantly more expensive than simple mono cameras.

The DE 10 2006 024 979 A1 discloses systems and methods for detecting an occupant within a vehicle, wherein depth information is obtained by means of two spaced-apart stereo cameras or a time-of-flight system.

This is where the invention comes in. The invention is based on the object of improving depth evaluations for localizing a vehicle occupant in a vehicle interior.

The object is achieved by a depth evaluation system for a vehicle for localizing a vehicle occupant in a vehicle interior with the features of claim 1. Furthermore, the object is achieved by a depth evaluation system for a vehicle for localizing a vehicle occupant in a vehicle interior with the features of claim 4 The object is achieved by a training method for an artificial neural network for calculating depth information with the features of claim 9. Furthermore, the object is achieved by a method for localizing a vehicle occupant in a vehicle interior with the features of claim 10. The object is also achieved by a computer program product for locating a vehicle occupant in a vehicle interior having the features of claim 12.

Further developments and advantageous configurations are given in the subclaims.

The depth evaluation device according to the invention is configured for a vehicle to localize a vehicle occupant in a vehicle interior. The depth evaluation device comprises at least one first input interface. First image data of a recording of the vehicle occupant are received via the first input interface. The depth evaluation device also includes an artificial neural network. The depth evaluation device is implemented, the artificial neural one Execute network when the artificial neural network is loaded in a memory of the depth evaluation device. The artificial neural network is designed to obtain first depth information from the first image data at least as a function of the first image data. The depth evaluation device is designed to obtain a first surface model of the vehicle occupant as a function of the first depth information. Furthermore, the depth evaluation device is designed to calculate the localization of the vehicle occupant in the vehicle interior as a function of the first surface model. The depth evaluation device also includes an output interface. At least the localization of the vehicle occupant is provided via the output interface.

Depth refers to the third dimension. Two-dimensional images have a height and a width. The depth appears as a third dimension through optical illusion, for example through the influence of light, color and / or line arrangement. The depth evaluation device according to the invention receives depth information for image points in the form of distances to object points. Depending on the distances obtained, a vehicle occupant can be localized relative to a reference point in the vehicle interior, for example to a camera that generates a picture of the vehicle occupant.

An evaluation device is a device that processes incoming information and outputs a result resulting from this processing. In particular, an electronic circuit, such as a central processor unit or a graphics processor, is an evaluation device.

A vehicle is in particular a land vehicle that is intended for operation on roads, for example a passenger car. The vehicle interior includes the enclosed space with its internal dimensions as well as the interior trim, seats, benches, the dashboard, other controls and the trunk. With regard to the triggering of vehicle occupant restraint systems, precise localization of the vehicle occupant in the vehicle interior is particularly necessary. For example, it is advantageous if an exact seating position of the vehicle occupant on a vehicle seat is known. The sitting position is obtained from the localization.

An interface is a device between at least two functional units at which an exchange of logical quantities, for example data, or physical quantities, for example electrical signals, takes place, either only unidirectionally or bidirectionally. The exchange can be analog or digital. The exchange can also be wired or wireless.

Image data is the data from an imaging sensor. For example, the color values for red, green and blue or gray levels of a pixel of the imaging sensor are image data. A pixel is a surface element of an imaging sensor. Brightness values are also image data. Infrared values are also image data. Color values, gray levels, brightness values and infrared values are two-dimensional image data. For example, a pixel contains a fixed number of maximum possible gradations for a color.

Two-dimensional then means that a respective color value is obtained for each individual gradation. For example, there can be 256 shades for the color red. With such a pixel, 256 levels of red can be displayed. Three-dimensional image data additionally contain a third item of information pixel by pixel, preferably the distance from image points to object points.

The first image data are preferably two-dimensional image data.

The depth information is preferably obtained for each pixel. The pixels of the imaging sensor are advantageously assigned to respective object classes as a function of the respective recording. The assignment of pixels to object classes is called semantic segmentation. An example of semantic segmentation for a motorcyclist is shown at the link http://blog.qure.ai/notes/semantic-segmentation-deep-learning-review. When recording a face of a vehicle occupant, object classes are, for example, eyes, nose and mouth.

Artificial intelligence is a generic term for the automation of intelligent behavior. For example, an intelligent algorithm learns to react appropriately to new information. An artificial neural network, also known as Artificial Neural Network, is an intelligent algorithm. An intelligent algorithm is designed to learn to react appropriately to new information. Automation of intelligent behavior is described by the term artificial intelligence.

In order to be able to react purposefully to new information, it is necessary that an artificial intelligence first learns the meaning of predetermined information. For this purpose, the artificial intelligence is trained with validation data. Validation data is generally a generic term for training data or test data. In particular, training data contain in addition to the actual Data also information about the meaning of the respective data. This training phase is inspired by the learning process of a brain. Training with validation data is called machine learning. A subset of machine learning is deep learning, so-called deep learning, in which a number of hierarchical layers of neurons, so-called hidden layers, are used to carry out the process of machine learning. Neurons are the functional units of an artificial neural network. An output of a neuron generally results as the value of an activation function evaluated using a weighted sum of the inputs plus a systematic error, the so-called bias. An artificial neural network with several hidden layers is a deep neural network.

In particular, the validation data is a data set with which the algorithm is tested during the development phase. Since the developer also makes decisions on the basis of the tests that influence the algorithm, another data set, the test data set, is used at the end of the development for a final, i.e. uninfluenced, evaluation.

A surface model represents the surface of an object, for example a body or body part, in three dimensions. For example, a surface model of a vehicle occupant's face represents an elevation of the nose, lips and / or skull bones. A surface model corresponds to a three-dimensional silhouette.

Since the depth information is obtained pixel by pixel, the surface model represents the surface of the vehicle occupant very precisely. Based on the knowledge of the precise surface of the vehicle occupant, distances between individual body parts of the vehicle occupant and a fixed reference point within the vehicle interior can be precisely determined. An exact localization of the vehicle occupant in the vehicle interior can thus be determined.

This localization is preferably provided to a vehicle occupant restraint system via the output interface in order to optimally parameterize the vehicle occupant restraint system. A vehicle occupant restraint system is a protective device used to mitigate the consequences of a collision or a traffic accident. Examples of vehicle occupant restraint systems are gas bag devices, referred to in English as airbags. The vehicle occupant restraint system preferably comprises interior and / or exterior airbags. Another example of a vehicle occupant restraint system is a belt tensioning device. For example, the ignition timing and / or ignition stages of a gas bag device are selected differently in the event that the tip of the nose of the vehicle occupant is located relatively close to a dashboard of the vehicle and / or at a relatively low height to the floor of the vehicle interior than in the event that the tip of the nose was located relatively far from the dashboard and / or at a relatively great height to the floor of the vehicle interior.

Depending on the surface model, a skeleton model and / or a degree of distraction is preferably obtained in order to parameterize the vehicle occupant restraint system in an improved manner. Localization of the vehicle occupant means here and below that, in particular, individual body parts and / or joints and / or their respective orientation are localized. The degree of distraction follows, for example, from a localization of the head, from a localization of an opening of the eyes and / or from a localization of the driver's line of sight of the vehicle occupant. In particular, when the eyes are opened, the driver's fatigue is recognized.

The first image data are preferably two-dimensional image data from a mono camera. Alternatively, the first image data are two-dimensional image data from a 3D camera. In a further embodiment of the invention, the first image data are three-dimensional image data from the 3D camera. The artificial neural network can calculate depth from the two-dimensional image data, which are then merged with the three-dimensional image data (depths) of the 3D camera in order to improve the three-dimensional image data. Further, the artificial neural network can calculate enhanced depth from the two-dimensional image data and the three-dimensional image data (depths), the two-dimensional image data and the three-dimensional image data being input data. In addition, the artificial neural network can only use the three-dimensional image data to improve them (gap filling / denoising). Furthermore, the artificial neural network can work on raw data from the camera to calculate depth information.

A mono camera is a monocular camera that includes a camera lens or a lens. A mono camera creates a two-dimensional image. A 3D camera is a camera that generates a three-dimensional image based on a functional principle. Examples of 3D cameras are stereo cameras and TOF cameras. Stereo cameras comprise two camera lenses or two objectives and generate a three-dimensional image from a combination of two simultaneously recorded two-dimensional images.

A mono camera has the advantage that it is comparatively compact, robust and inexpensive. With the depth evaluation device according to the invention, depth information is obtained with the artificial neural network. The artificial neural network has the two-dimensional image data from the mono camera as input and outputs a depth image.

The artificial neural network preferably comprises convolutional layers comprising a plurality of convolutional and / or pooling layers known to the person skilled in the art. In a convolutional neural network, a filter is applied to a layer of neurons regardless of the position with the same weights.

Image data from 3D cameras are often incomplete. For example, image data from stereo cameras are incomplete and therefore incomplete due to hidden areas at the edges of objects or objects with little texture. Due to the poorly reflective surfaces, image data from TOF cameras are incomplete. In addition to three-dimensional image data, a 3D camera also provides two-dimensional mono or stereo images. In one embodiment of the invention, the artificial neural network is trained on these two-dimensional image data from the 3D camera. As a result, depth information can be calculated independently of a functional principle of the 3D camera. According to the invention, this calculated depth information is merged with the depth information calculated according to the functional principle of the 3D camera. Alternatively, the artificial neural network can use the incomplete three-dimensional image data directly as input values, preferably with or without additional two-dimensional image data. This trains the artificial neural network to generate refined three-dimensional image data.

So that the mono camera and / or the 3D camera can also capture the vehicle interior at night, active lighting of the vehicle interior is advantageous, for example by means of a projector that illuminates the vehicle interior with infrared light. In this case, the mono camera and / or the 3D camera are infrared cameras for detecting reflected infrared light.

The projector is designed to illuminate the vehicle interior evenly. Alternatively, the projector is designed to project a beam pattern into the vehicle interior. Depth information is obtained as a function of a geometric change in the beam pattern reflected on an object in the vehicle interior compared to a known static beam pattern. Depending on the extent and distance of the object, patterns of reflected radiation patterns appear distorted in comparison to patterns of transmitted static radiation patterns. For example, points of a transmitted point pattern appear distorted on a distant curved surface compared to a nearby flat surface. This means that the distance and thus the depth information to the object is determined as a function of this geometric change. A ray pattern is a geometric form of lighting that is achieved with these rays through a corresponding radiator that generates these patterns. Patterns are, for example, geometric shapes such as lines or circles. Pattern also means a sequence of such shapes. The patterns are achieved by spatial coding of signals.

The mono camera and / or the 3D camera capture and evaluate preferably infrared light. The mono camera and / or the 3D camera, infrared cameras are particularly preferred. An infrared camera can also detect vehicle occupants via their heat radiation, so that a projector is optional.

According to the invention, the depth evaluation device comprises a second input interface. Second image data of a recording of the vehicle occupant are received via the second input interface. The first image data are the image data of a mono camera. The second image data are the image data of a further camera, preferably a 3D camera. The artificial neural network is designed to obtain second depth information from the second image data as a function of the second image data. The depth evaluation device is designed to obtain a second surface model of the vehicle occupant as a function of the second depth information. Depending on the second surface model, the depth evaluation device calculates the localization of the vehicle occupant in the vehicle interior. The depth evaluation device also includes a logic module. The logic module is designed to merge the first surface model with the second surface model. The depth evaluation device is designed to calculate the localization of the vehicle occupant in the vehicle interior as a function of this fusion.

A logic module is a device that processes input signals into an output signal, the input signals being converted into a single logical result by implementing logical operators such as AND, OR, NOT, in order to obtain the output signal. Logic modules are in particular in the form of digital integrated circuits, for example simple AND, OR, NOT circuits or programmable elements such as Example Field Programmable Gate Array, abbreviated FPGA, or Application Specific Integrated Circuits, abbreviated ASICs, implemented. For example, an artificial neural network can be implemented on an FPGA.

Through the merger, data sets, especially incomplete data sets, are completed and thus improved information is obtained.

In particular, redundancy is achieved when using a mono camera and a 3D camera. A mono camera is preferably used to calculate the information obtained by a 3D camera about the surface description of the vehicle occupant in a path independent of the 3D camera, namely by outputting depth information from two-dimensional image data of the mono camera according to the invention. As a result, the mono camera provides surface information even if the 3D camera is covered or fails, for example. Furthermore, the functioning of the 3D camera can be checked as a function of the depth information obtained with the mono camera and the functioning of the depth evaluation device can be checked as a function of the depth information obtained with the 3D camera.

In an advantageous embodiment of the invention, the depth evaluation device comprises at least one third input interface. The third input interface is used to receive seat occupancy data about vehicle seat occupancy of the vehicle occupant. The artificial neural network is designed to receive the first and / or second depth information as a function of the seat occupancy data and / or the first and / or second image data. The depth evaluation device is preferably designed to calculate a seat model as a function of the seat occupancy data and to project the seat model into the receptacle of the vehicle occupant.

Seat occupancy data are recorded with weight sensors, for example. The weight sensors are preferably arranged distributed over a vehicle seat. Strain gauges, for example, which can preferably be arranged in the form of foils on vehicle seats, are suitable as weight sensors. Seat occupancy data are also recorded via the adjustment positions of the vehicle seat or via belt extension lengths. Seat occupancy data also includes seat settings. The seat occupancy data, as well as e.g. also belt extension length, can also be further input variables for the artificial neural network.

Detectable features are advantageously attached to the belt. The first and / or the second image data comprise these features. The artificial neural network is thus fed with these features. Depending on these features, the artificial neural network learns to better recognize the position of the belt. Furthermore, the artificial neural network learns the appearance of these features for different distances and / or curvatures. The features are preferably reflectors that reflect only in the infrared spectrum. Such features are not noticeable in the optically visible spectrum.

The seat model is in particular a virtual three-dimensional model of a vehicle seat on which the vehicle occupant is sitting. The vehicle seat itself, as well as other static objects, can be projected very precisely into the image. In this way, an independent depth image is generated or simulated, in this case without a person. Above all, this could serve as an additional input for the neural network in order to better estimate the final depth, or it could be used as a second variant for merging afterwards. By projecting the seat model into the receptacle of the vehicle occupant, a maximum possible depth assumption is advantageously made for the pixels in the area of the vehicle seat. The depth evaluation device is preferably designed to convert a maximum assumed depth into an absolute depth estimate by means of features on the seat that are specifically detected, ie not covered, in the receptacle of the vehicle occupant. Such projections of known objects can again be used for fusion or as. For example, the artificial neural network will learn the distance of the head in the two-dimensional image to the recording camera from a comparison of a head size from a two-dimensional image recording to the head size at a predetermined distance.

The depth evaluation system according to the invention is configured for a vehicle for localizing a vehicle occupant in a vehicle interior. The depth evaluation system comprises at least one mono camera and a depth evaluation device according to the invention. The localization of the vehicle occupant in the vehicle interior is thus calculated. A data exchange between the mono camera and the depth evaluation device takes place via the first input interface.

The depth evaluation system advantageously comprises at least one 3D camera. A data exchange between the 3D camera and the depth evaluation device takes place via the second input interface. The mono camera can preferably be arranged on a steering wheel of the vehicle and / or the 3D camera can be arranged on a windshield, preferably on a strip of the windshield of the vehicle.

According to a development of the invention, the depth evaluation system comprises a seat occupancy sensor. The seat occupancy sensor detects seat occupancy and outputs seat occupancy data on vehicle seat occupancy of the vehicle occupant. A data exchange between the seat occupancy sensor and the depth evaluation device takes place via the third input interface.

The seat occupancy sensor is, for example, a weight sensor. According to the invention, several seat occupancy sensors are arranged distributed over a vehicle seat. The seat occupancy sensor is alternatively a strain gauge. A plurality of strain gauges in the form of foils can preferably be arranged on vehicle seats. Alternatively, the seat occupancy sensor is a sensor that detects an adjustment position of the vehicle seat. According to a further alternative, the seat occupancy sensor is a sensor that detects a belt extension length. In a further embodiment of the invention, the seat occupancy data are recorded by a combination of the aforementioned seat occupancy sensors.

In a preferred embodiment of the invention, the depth evaluation system comprises a vehicle occupant restraint system. A data exchange between the vehicle occupant restraint system and the depth evaluation device takes place via the output interface. With the data exchange, parameters of the vehicle occupant restraint system are adapted to the localization of the vehicle occupant.

In a further embodiment of the invention, the depth evaluation system comprises a display device. The display device is designed to display the first surface model, the second surface model, the surface model merged from the first and second surface model and / or the location of the vehicle occupant. The display device is designed, for example, as a display of an infotainment system.

According to the invention, a vehicle comprises a depth evaluation system according to the invention. In particular, the vehicle can be operated in an automated manner. In order to achieve higher levels of automation, it is necessary to know exactly how distracted people are, fatigue and, in particular, where they are located in the vehicle interior. This is achieved with the depth evaluation device according to the invention.

A vehicle that can be operated automatically is a vehicle that has technical equipment that enables the respective vehicle to cope with a driver's task, including longitudinal and lateral guidance, after activation of a corresponding automatic driving function, in particular a highly or fully automated driving function according to the SAEJ3016 standard a vehicle control device can control.

A partially automated vehicle can take over certain driving tasks. The SAEJ3016 standard differentiates between the automation levels SAE Level 2 and SAE Level 3. Level 2 is defined by the fact that the driving mode-specific execution of steering and acceleration / braking processes is carried out by one or more driver assistance systems using information about the driving environment the expectation that the human driver will perform all remaining aspects of the dynamic driving task. Level 3 is defined by the fact that the driving mode-specific execution of all aspects of the dynamic driving task is carried out by an automated driving system with the expectation that the human driver will react appropriately to the system's request.

A fully automated vehicle replaces the driver. The SAEJ3016 standard differentiates between SAE Level 4 and SAE Level 5 Level 4 is defined in that the driving mode-specific execution of all aspects of the dynamic driving task is carried out by an automated driving system, even if the human driver does not react appropriately to the system's request. Level 5 is defined in that the continuous execution of all aspects of the dynamic driving task is carried out by an automated driving system under all driving and environmental conditions that can be managed by a human driver.

A pure assistance system assists the driver in performing a driving task. This corresponds to SAE level 1.

The invention is intended in particular for SAE levels 3, 4 and 5. In particular, in a transition period to fully automated driving, the invention is used at SAE levels 3 and 4 in order to then be used at SAE level 5.

The training method according to the invention trains an artificial neural network for calculating depth information. In a first process step, training data are obtained. The training data include at least two-dimensional image data from a mono camera of a recording of the vehicle occupant. Furthermore, the training data include first target depth information belonging to the respective recording. In addition or as an alternative, the training data include seat occupancy data relating to the vehicle seat occupancy of the respective recording Vehicle occupants. In a second process step, the artificial neural network is fed forward with the training data. In a third method step, first actual depth information calculated with the artificial neural network is obtained in this forward feed. In a fourth method step, weights for connections of neurons of the artificial neural network are obtained by minimizing an error between the first actual depth information and the first target depth information in a backward feed of the artificial neural network. The artificial neural network is thus trained to obtain first depth information from the at least first image data.

Forward feeding, referred to as forward propagation, means that information is fed into the input layer of the artificial neural network, passes through the following layers and is output in the output layer. Backward feeding, referred to in English as backward propagation, means that information, in particular an error or a gradient of an error, is input into the output layer and output in the input layer. By successively feeding back a received output error from the output layer into the respective previous layer up to the input layer, the errors of the respective layers are obtained. The errors are a function of the weights. By minimizing the error in the training phase, the weights are changed. As a result, when the input is fed in again, an approximation of the desired output is achieved.

Weights are factors used to define connections between neurons. If a weight takes on the value zero, the neuron on the output side of the associated connection is switched off, i.e. it receives no input and does not perform any task. But a neuron has more than one connection. In general, a weight of zero means that the neuron on the input side of the connection has no influence on the activation of the neuron on the output side. With a higher value of the weight, this neuron is activated correspondingly more strongly.

The artificial neural network learns to recognize distance information in 2D images.

The first image data are alternatively two-dimensional image data from a 3D camera. After the training, the artificial neural network learned to also process three-dimensional image data.

The method according to the invention localizes a vehicle occupant in a vehicle interior. In a first method step, first image data of a picture of the vehicle occupant are obtained. In a second process step, an artificial neural network is fed forward with the first image data. The artificial neural network is trained according to the training method according to the invention. In a third method step, first depth information is obtained as a function of the first image data in this forward feed. In a fourth method step, a first surface model of the vehicle occupant is obtained as a function of the first depth information. In a fifth method step, the localization of the vehicle occupant in the vehicle interior is calculated as a function of the first surface model.

The first image data are preferably two-dimensional image data from a mono camera. The localization is preferably provided via an output interface.

According to the invention, second image data are obtained. The artificial neural network is fed forward with the second image data. Second depth information is obtained in this forward feed as a function of the second image data. A second surface model of the vehicle occupant is obtained as a function of the second depth information. The second surface model is fused with the first surface model. The localization of the vehicle occupant is calculated as a function of this fusion. Preferably, when at least one of the surface models is obtained, seat occupancy data on vehicle seat occupancy of the vehicle occupant are merged with the respective depth information.

The second image data are preferably two-dimensional or three-dimensional image data from a 3D camera. By merging the first surface model and the second surface model, redundancy is achieved in particular for the 3D camera. The surface of the vehicle occupants is also described using the first image data and thus in an independent path to the second image data.

A seat model is advantageously calculated as a function of the seat occupancy data. The seat model is preferably projected into the receptacle for the vehicle occupant. By projecting the seat model into the receptacle of the vehicle occupant, a maximum possible depth assumption is advantageously made for the pixels in the area of the vehicle seat.

The localization of the vehicle occupant is preferably a vehicle occupant restraint system, a distraction detection system and / or a drowsiness detection system. Parameters of the vehicle occupant restraint system, the distraction detection system and / or the drowsiness detection system are adapted to the localization of the vehicle occupant. Thus, in a vehicle occupant restraint system, ignition times and / or ignition stages of gas bag devices and / or belt tensioning lengths are optimally adapted to the vehicle occupant in a given collision situation.

According to the invention, a depth evaluation device according to the invention or a depth evaluation system according to the invention is used to carry out the method.

The computer program product according to the invention is configured to locate a vehicle occupant in a vehicle interior. The computer program product is designed to be loaded into a memory of a computer. The computer program product comprises software code sections with which the method steps of the method according to the invention are carried out when the computer program product is running on the computer.

A computer is a data processing system. Software is a collective term for programs and associated data. The complement to software is hardware. Hardware describes the mechanical and electronic alignment of a data processing system.

A memory is a medium for the backup of data. The memory contains the programs or program parts to be executed and the data required for them. The memory is, for example, a reversed part of a main memory of the computer. Alternatively, the memory is a memory module. A main memory, for example, can be expanded with a memory module.

Computer program products generally comprise a sequence of instructions which, when the program is loaded, cause the hardware to carry out a specific method that leads to a specific result. When the relevant program is used on a computer, the computer program product produces a technical effect, namely the localization of a vehicle occupant in a vehicle interior. In particular, vehicle occupant restraint systems are optimally parameterized as a function of this localization. This parameterization is preferably carried out by the computer program product.

The computer program product according to the invention is platform independent. That means it can run on any computing platform. The computer program product is preferably executed on a depth evaluation device according to the invention.

• 1 ein Ausführungsbeispiel eines Fahrzeuges mit einer erfindungsgemäßen Tiefenauswerteeinrichtung und einem erfindungsgemäßen Tiefenauswertungssystem,
• 2: eine schematische Darstellung des erfindungsgemäßen Trainingsverfahrens und
• 3 eine schematische Darstellung des erfindungsgemäßen Verfahrens.

The invention is illustrated by way of example in the accompanying figures. Show it:

• 1 an embodiment of a vehicle with a depth evaluation device according to the invention and a depth evaluation system according to the invention,
• 2 : a schematic representation of the training method according to the invention and
• 3 a schematic representation of the method according to the invention.

1 shows a vehicle 1 . The vehicle 1 is a passenger car. In a vehicle interior 3 of the vehicle sits a vehicle occupant 2 on a vehicle seat in the position of a vehicle driver.

The vehicle 1 includes an in-depth evaluation system 20th . The depth evaluation system 20th includes a mono camera 4th . The mono camera 4th takes two-dimensional images of the vehicle occupant 2 with a security level of at least ASIL B. ASIL stands for Automotive Safety Integrity Level and designates safety levels according to the ISO 26262 standard. The safety levels differ in the required failure probability. With ASIL B, the recommended failure probability is less than 10-7 per operating hour. The mono camera 4th is on a steering wheel 6th arranged. The mono camera 4th has a wide field of vision. From the recordings of the vehicle occupant 2 generates the mono camera 4th first, two-dimensional, image data.

The depth evaluation system 20th also includes a 3D camera 5. The 3D camera 5 takes three-dimensional recordings of the vehicle occupant 2 with a security level of at least ASIL B. The 3D camera 5 is a TOF indoor camera. The 3D camera is on a windshield 7th of the vehicle 1 arranged. From the recordings of the vehicle occupant 2 the 3D camera generates second, three-dimensional, image data.

Alternatively, there is another mono camera 4th provided, for example, on the windshield 7th of the vehicle 1 is arranged. This more mono camera 4th can replace the 3D camera 5 or be provided in addition to the 3D camera 5. Further 3D cameras 5 can also be provided.

The depth evaluation system also includes 20th a depth evaluation device 10 . The depth evaluation device 10 is for example a Chip. The depth evaluation device 10 is implemented as a system-on-a-chip, i.e. all or at least a large part of the functions are integrated on one chip. The chip comprises, for example, a multi-core processor with at least six central processing processors, referred to as Central Processing Unit, abbreviated CPU. The chip also includes at least 256 graphics processors, referred to as Graphic Processing Unit, or GPU for short. The depth evaluation device has such a structure 10 scalable, i.e. the depth evaluation device 10 can be customized for different SAE levels. This is because higher automation levels require more computing power than lower automation levels.

The depth evaluation device 10 comprises a first input interface 11 . Via the first input interface 11 receives the depth evaluation device 10 the first image data from the mono camera 4th . The depth evaluation device 10 also includes a second input interface 12th . Via the second input interface 12th receives the depth evaluation device 10 the second image data 3D camera 5.

The depth evaluation device also includes 10 a third input interface 13th . Via the third input interface 13th receives the depth evaluation device 10 Seat occupancy data from a seat occupancy sensor 21st . The seat occupancy sensor 21st is a weight sensor. Alternatively or additionally, the depth evaluation device is provided 10 via the third input interface 13th Data on a belt extension length.

The first, second and / or third input interface 11 , 12th , 13th are preferably interfaces in an on-board network of the vehicle 1 .

The depth evaluation device 10 includes a memory 16 . In the memory 16 is an artificial neural network 14th loaded. The depth evaluation device 10 leads the artificial neural network 14th out.

Neurons 15th an input layer of the artificial neural network 14th receive the first and second image data and the seat occupancy data as input signals. Alternatively, the artificial neural network processes 14th first the first image data and then the further data. In addition, another artificial neural network 14th be provided in order to process the further data, that is to say the second image data and / or the seat occupancy data. The artificial neural network 14th exhibits an architecture of neurons 15th , Connections between neurons and layers of neurons, with which first and second depth information can be obtained particularly effectively from the input signals. This includes the artificial neural network 14th multiple layers of folding and pooling. The first and second depth information is in an output layer of the artificial neural network 14th provided as its output signals.

The artificial neural network calculates the first depth information 14th depending on the first, two-dimensional, image data from the mono camera 4th . The artificial neural network is trained accordingly for this purpose. The artificial neural network calculates the second depth information 14th as a function of the second, three-dimensional, image data of the 3D camera 5. This calculation improves the depth information already obtained with the 3D camera 5.

The artificial neural network is trained accordingly for this purpose. The artificial neural network is used to improve the depth assumptions 14th the seat occupancy data.

Depending on the first and second depth information, the depth evaluation device calculates a first and a second surface model of the vehicle occupant 2 . A logic module 18th merges the first and second surface models. Alternatively, the logic module checks 18th whether the first surface model is redundant to the second surface model. The logic module 18th is integrated, for example, on the chip of the depth evaluation device. The logic module is, for example, an FPGA. The surface model obtained from this fusion is transmitted via an output interface 17th the depth evaluation device to a vehicle occupant restraint system 22nd provided. The vehicle occupant restraint system 22nd is includes, for example, an actuator for belt length adjustment. Depending on the surface model, the actuator adjusts the belt length optimally to the surface of the vehicle occupant 2 to adjust.

The depth evaluation device 10 is implemented in particular only from the mono camera using the first, two-dimensional image data 4th calculated first depth information a first surface model for localizing the vehicle occupant 2 to provide. By generating the first depth information, it is possible to use a surface model of the vehicle occupant for further processing, as in the case of a TOF or stereo camera 2 to generate, and thereby also to calculate a skeleton model, degree of distraction and localization of the vehicle occupant.

The training method according to the invention for the artificial neural network 14th is in 2 shown.

In one process step T1 training data are obtained. The training data are, for example, at least two-dimensional image data from the mono camera 4th and first target depth information associated with the respective recording.

In one process step T2 the artificial neural network is fed forward with the training data.

This feed forward is made with the artificial neural network 14th calculated first actual depth information in a third method step T3 receive.

In a fourth process step T4 become weights for connections of neurons 15th of the artificial neural network 14th by minimizing an error between the first actual depth information and the first target depth information in a reverse feed of the artificial neural network 14th receive. The backward feed gives the gradient of the error as a function of the weights and thus a weight change that should minimize the error. By repeating the procedure from step T1 , in particular with a potentially different subset of the training data, the error is iteratively reduced more and more. The validation data is used over and over again after a few iterations to check the error for data that has not been seen. For example, you can decide to end the training early because the error in the validation data may grow again, since in that case the network “learns by heart” the training data and overfitting occurs.

With the training method, the artificial neural network becomes 14th trained to obtain first depth information from the at least first image data.

The procedure for locating the vehicle occupant 2 in the vehicle interior 3 is in 3 shown.

In one process step V1 are first image data of a recording of the vehicle occupant 2 receive. The first image data are for example with the mono camera 4th receive. Second image data are obtained in a method step V1a. The second image data are obtained, for example, with the 3D camera.

In one process step V2 becomes the artificial neural network 14th fed forward with the first image data. In time before, in parallel or afterwards, the artificial neural network is fed forward with the second image data in a method step V2a. The artificial neural network 14th is an artificial neural network trained according to the training method according to the invention.

In this feed forward are in one process step V3 receive first depth information as a function of the first image data. Correspondingly, in a method step V3a, second depth information is obtained as a function of the second image data.

In one process step V4 becomes a first surface model of the vehicle occupant depending on the first depth information 2 . Accordingly, in a method step V4a, a second surface model of the vehicle occupant is generated as a function of the second depth information 2 receive.

In one process step V5 the localization of the vehicle occupant is dependent on the first surface model 2 in the vehicle interior 3 receive.

The second surface model is combined with the first surface model in one process step V6 merged.

The localization of the vehicle occupant 2 is dependent on this merger in one process step V7 calculated.

In one process step V8 when at least one of the surface models is obtained, seat occupancy data of the seat occupancy sensor 21st via a vehicle seat occupancy of the vehicle occupant 2 read in to improve a depth estimate. The seat occupancy data can also be merged with the respective depth information.

The localization of the vehicle occupant 2 is in one process step V9 the vehicle occupant restraint system 22nd provided. Vehicle occupant restraint system parameters 22nd are in one process step V10 the localization of the vehicle occupant 2 customized.

Claims (12)

Depth evaluation device (10) for a vehicle (1) for localizing a vehicle occupant (2) in a vehicle interior (3) comprising • a first input interface (11) to receive first image data of a recording of the vehicle occupant (2), • a second input interface ( 12) in order to obtain second image data of a recording of the vehicle occupant (2), • the first image data being the image data of a mono camera (4) and the second image data being the image data of a 3D camera (5), • an artificial neural network (14), o the depth evaluation device (10) being implemented, the artificial neural network (14) to be carried out when the artificial neural network (14) is loaded in a memory (16) of the depth evaluation device (10), o the artificial neural network (14) being implemented, depending on the first image data, first depth information from the first image data and to obtain second depth information from the second image data as a function of the second image data, wherein the depth evaluation device (10) is designed to obtain a first surface model of the vehicle occupant (2) as a function of the first depth information, o as a function of the second depth information to obtain a second surface model of the vehicle occupant (2) depending on the first surface hen model to calculate the localization of the vehicle occupant (2) in the vehicle interior (3), o to calculate the localization of the vehicle occupant (2) in the vehicle interior (3) as a function of the second surface model, o wherein the depth evaluation device (10) includes a logic module (18 ), which is designed to merge the first surface model with the second surface model, o wherein the depth evaluation device (10) is designed to calculate the localization of the vehicle occupant (2) in the vehicle interior (3) as a function of this merger and • an output interface (17) in order to provide at least the localization of the vehicle occupant (2). Depth evaluation device (10) Claim 1 wherein the first image data are two-dimensional image data of a 3D camera (5) or three-dimensional image data of the 3D camera (5). Depth evaluation device (10) Claim 1 , • wherein the depth evaluation device (10) comprises at least one third input interface (13) in order to receive seat occupancy data about vehicle seat occupancy of the vehicle occupant (2), and • wherein the artificial neural network (14) is implemented, the first and / or second depth information as a function of the seat occupancy data and / or the first and / or second image data, the depth evaluation device (10) being designed to calculate a seat model as a function of the seat settings and to project the seat model into the receptacle of the vehicle occupant (2). Depth evaluation system (20) for a vehicle (1) for localizing a vehicle occupant (2) in a vehicle interior (3) comprising • at least one mono camera (4) and • a depth evaluation device (10) according to one of the Claims 1 to 3 to calculate the localization of the vehicle occupant (2) in the vehicle interior (3), data being exchanged between the mono camera (4) and the depth evaluation device (10) via the first input interface (11). Depth evaluation system (20) Claim 4 , • wherein the depth evaluation device (10) has a depth evaluation device (10) Claim 1 or 3 is, • wherein the depth evaluation system (20) comprises at least one 3D camera (5), o wherein a data exchange between the 3D camera (5) and the depth evaluation device (10) takes place via the second input interface (12), and • wherein the Mono camera (4) can be arranged on a steering wheel (6) of the vehicle (1) and / or the 3D camera (5) can be arranged on a windshield (7) of the vehicle (1). Depth evaluation system (20) Claim 4 or 5 , • wherein the depth evaluation device (10) has a depth evaluation device (10) Claim 3 • wherein the depth evaluation system (20) comprises a seat occupancy sensor (21) in order to obtain seat occupancy data on vehicle seat occupancy of the vehicle occupant (2), data being exchanged between the seat occupancy sensor (21) and the depth evaluation device via the third input interface (13). Depth evaluation system (20) according to one of the Claims 4 to 6th , wherein the depth evaluation system (20) comprises a vehicle occupant restraint system (22) and an exchange of data between the vehicle occupant restraint system (22) and the depth evaluation device (10) takes place via the output interface (17) in order to determine parameters of the vehicle occupant restraint system (22) to adapt to the localization of the vehicle occupant (2). Depth evaluation system (20) according to one of the Claims 4 to 7th wherein the depth evaluation system (20) comprises a display device to display the first surface model, the second surface model, the surface model merged from the first and second surface model and / or the location of the vehicle occupant (2). Training method for an artificial neural network (14) for calculating depth information comprising the following method steps: Obtaining training data, the training data being at least two-dimensional image data from a mono camera (4) of a recording of the vehicle occupant (2), first target values belonging to the respective recording Depth information and / or seat occupancy data belonging to the respective recording about vehicle seat occupancy of the vehicle occupant (2) include (T1), • Feeding the artificial neural network forward with the training data (T2), • Receiving this forward feed with the artificial neural network (14) calculated first actual depth information (T3) and • obtaining weights for connections of neurons (15) of the artificial neural network (14) by minimizing an error between the first actual depth information and the first target depth information in a backward feed of the artificial neural network rk (14) (T4), in order to train the artificial neural network (14) to obtain first depth information from the at least first image data. A method for localizing a vehicle occupant (2) in a vehicle interior (3) comprising at least the following method steps: • obtaining first image data of a recording of the vehicle occupant (2 (V1) • forwarding an artificial neural network (14) with the first image data, wherein the artificial neural network (14) after the training process Claim 10 is trained (V2), • in this feed forward, receiving first depth information as a function of the first image data (V3), • as a function of the first depth information, receiving a first surface model of the vehicle occupant (2) (V4), • as a function of the first surface model, calculating the Localization of the vehicle occupant (2) in the vehicle interior (3) (V5). where • second image data are obtained (V1a), • the artificial neural network (14) is fed forward with the second image data (V2a), • second depth information is obtained in this forward feed as a function of the second image data (V3a), • as a function of the second depth information a second surface model of the vehicle occupant (2) is obtained (V4a), • the second surface model is fused with the first surface model (V6) and • the localization of the vehicle occupant (2) is calculated as a function of this fusion (V7), • where When at least one of the surface models is obtained, seat occupancy data about vehicle seat occupancy of the vehicle occupant are read in in order to improve a depth estimate (V8), wherein a depth evaluation device (10) according to one of the Claims 1 to 3 or a depth evaluation system (20) according to one of the Claims 4 to 8th is used. Procedure according to Claim 10 , the localization of the vehicle occupant (2) being provided to a vehicle occupant restraint system (22), a distraction detection system and / or a drowsiness detection system (V9) and parameters of the vehicle occupant restraint system (22), the distraction detection system and / or the drowsiness detection system to the localization of the Vehicle occupants (2) are adapted (V10). Computer program product for locating a vehicle occupant (2) in a vehicle interior (3), wherein the computer program product • is designed to be loaded into a memory of a computer and • comprises software code sections with which the method steps of the method according to one of the Claims 10 to 11 be executed when the computer program product is running on the computer.

Images