DE102022123465A1 - METHOD AND SYSTEM FOR TRAINING AN EVALUATION MODEL FOR DETECTING A SEAT OCCUPANCY STATE OF A SEAT ARRANGEMENT BASED ON RADAR POINT CLOUDS - Google Patents

Abstract

A method for training an evaluation model, in particular a machine learning model, for an automated recognition of a seat occupancy status of a seating arrangement with at least one seat has: (i) acquiring measurement data which represent an associated radar point cloud (705), wherein the radar point cloud ( 705) was or is obtained on the basis of a radar scan of a spatial area surrounding the seat arrangement at least in sections and is assigned to one of several predefined possible seat occupancy states of the seat arrangement; (ii) generating augmented measurement data, which represents an associated augmented point cloud (710), from the measurement data associated with the radar point cloud (705); and (iii) generating training data from the measurement data and the augmented measurement data, the training data being provided as input data to the evaluation model in order to obtain as its output an evaluation result which is assigned to the seat occupancy status of the seating arrangement. Generating the augmented measurement data includes: (i) determining multiple anchor points within the spatial region; (ii) determining a transformation with respect to each of the anchor points, each of the transformation being applied to the points (715) of the radar point cloud (705) to obtain a transformed point cloud associated with the respective anchor point; and (iii) merging the transformed point clouds associated with the anchor points to obtain the augmented point cloud (710) associated with the augmented measurement data. A method for automatically recognizing a seat occupancy state, in particular seat-related, of a seating arrangement with at least one seat uses the trained evaluation model.

Description

The present invention relates to a method, a computer program and a system configured to carry out the method, each for automatically recognizing a seat occupancy status of a seating arrangement with at least one seat using a trained evaluation model. In particular, the invention also relates to a method for training such an evaluation model, in particular a machine learning model.

In various situations it may be necessary to automatically determine the current seat occupancy status of a seating arrangement with at least one seat. Such a situation can occur in particular in vehicles, for example in motor vehicles, where a configuration of the vehicle or an activation, deactivation and/or control of one or more vehicle functionalities should take place depending on a current seat occupancy status. For example, in motor vehicles it is known to issue an acoustic or visual indication to vehicle occupants to fasten seat belts or to control the activation or deactivation of airbags depending on a detected seat occupancy status.

For the automated detection of a current seat occupancy status of one or more seats, in particular an arrangement of vehicle seats in a vehicle, so-called seat occupancy mats are known in particular, which are integrated into the seats (usually one per seat) and pressure-sensitive sensors for detecting occupancy of the respective seat use. A seat occupancy status of the seat is determined depending on the sensor signals or sensor data from these sensors, usually by means of a threshold value test.

These known solutions therefore require the seats to be equipped with sensors built into them and are also usually unable to distinguish between different seat occupancy states beyond “occupied” and “unoccupied”. Furthermore, the sensors typically have to be integrated into the seats at the factory, so that retrofitting is difficult or impossible and there is no need to record the seat occupancy status of seats that are not equipped in this way.

Methods are therefore also known which use radar technology to automatically detect a current seat occupancy status. By scanning the vehicle interior using radar sensors, measurement data is generated in the form of radar point clouds. Based on the measured radar point cloud, current seat occupancy can be determined. For this purpose, a corresponding evaluation model can be used, which can be a machine learning model. This model can be trained with appropriate data before actual use. However, in order to obtain a reliable result, a considerable amount of training data must be available, which in turn requires considerable effort.

It is an object of the present invention to provide an improved solution for automatically recognizing a seat occupancy status of a seating arrangement with at least one seat. In particular, an improved solution for training an evaluation model for automated recognition of a seat occupancy status of a seating arrangement is to be specified.

The solution to this problem is achieved in accordance with the teaching of the independent claims. Various embodiments and developments of the invention are the subject of the subclaims.

A first aspect of the solution presented here relates to a method, in particular a computer-implemented one, for training an evaluation model, in particular a machine learning model, for automated recognition of a seat occupancy status of a seating arrangement with at least one seat. The method comprises: (i) acquiring, in particular receiving or generating, measurement data which represent an associated radar point cloud, wherein the radar point cloud was or is obtained on the basis of a radar scanning of a spatial area surrounding the seat arrangement at least in sections and one of several predefined possible seat occupancy states assigned to the seating arrangement; (ii) generating augmented measurement data representing an associated augmented point cloud from the measurement data associated with the radar point cloud; and (iii) generating training data from the measurement data and the augmented measurement data, the training data being provided as input data to the evaluation model in order to obtain as its output an evaluation result which is assigned to the seat occupancy status of the seating arrangement. Generating the augmented measurement data includes: (i) determining multiple anchor points within the spatial region; (ii) determining a transformation with respect to each of the anchor points, each of the transformation being applied to at least a portion of the points of the radar point cloud to obtain a transformed point cloud associated with the respective anchor point; and (iii) merging the associated anchor points th transformed point clouds to obtain the augmented point cloud, which is assigned to the augmented measurement data.

Using the method according to the first aspect, an evaluation model, which can in particular be a machine learning model, can be effectively trained, so that the accuracy in recognizing a seat occupancy status can be improved. With the help of data augmentation, an increased amount of training data can be easily generated without the need for new actual measurement data. The training method is particularly advantageous for the machine learning of a radar-based system for recognizing seat occupancy. The method involves training for machine learning using actually measured radar point clouds and augmented point clouds in parallel. By augmenting (also called “expanding”) the points from different anchor points or starting points, multiple sets of augmented (expanded) point clouds can be easily obtained. These augmented point clouds are stitched together to obtain the additional training data.

The term “seat occupancy status” of a seating arrangement with at least one seat, as used herein, is to be understood in particular as meaning information which indicates whether or to what extent the seating arrangement or at least one of its seats is occupied by an object, in particular a thing or a person . is occupied. In a simple example, the seat occupancy status can only indicate the presence or absence of an object, or in a more developed example, in the case of the presence of at least one object on the seating arrangement or one or more of its seats, a statement about the type or another Specify a property, such as a spatial extent, of the object.

The term “radar point cloud”, as used herein, is to be understood in particular as meaning a set of points of a vector space obtained by means of radar scanning of at least one object surface, which has a, typically unorganized, spatial structure (“cloud”). In the case of a radar point cloud, the points of the radar point cloud may be referred to as “radar points”. A (radar) point cloud can be described in particular by the (radar) points it contains. The radar points in turn can each be described in particular by their spatial coordinates, which indicate for each radar point a location of reflection of an emitted radar signal on an object surface, measured during radar scanning. Additional attributes such as: B. measured Doppler velocity or a signal-to-noise ratio (SNR) can be recorded.

The term “augmentation” or “data augmentation” as used herein means an expansion or enlargement of a data set that is used for training purposes for an evaluation model, in particular a machine learning model. Data augmentation is used to increase the amount of data by adding slightly modified copies of existing data or newly created synthetic data from existing data. It also acts as a regulator when training a machine learning model and helps reduce overfitting when training a machine learning model.

The term “augmented point cloud” as used herein means in particular a point cloud that was obtained through data augmentation of a radar point cloud.

The term “anchor point” as used herein is to be understood in particular as a starting point or reference point that is used for the points of the point cloud to be transformed with regard to a transformation of a point cloud. For example, an anchor point can serve as a reference point for a measurement, such as a distance measurement, as a center for a transformation, such as a center of rotation for a rotation. The anchor point can form a zero point of a coordinate system or space in which the transformation is carried out. The anchor point can also be subject to the transformation itself.

The term “evaluation model”, as used herein, is to be understood as meaning a model, in particular a mathematical one, which uses a radar point cloud or one or more parameters characterizing it as input variable(s) in order to deliver an evaluation result depending on this, in the present case one of several predefined possible seat occupancy states of the seating arrangement. The evaluation model can in particular be a mathematical estimation function, with the radar point cloud representing empirical data as a sample and the evaluation result representing an estimated value determined depending on this. The evaluation model can in particular be a “machine learning model” (or equivalently “machine learning model”), which here includes in particular a mathematical model created using at least one machine learning algorithm on the basis of example data, which is referred to as training data. especially statistical,model for making predictions or decisions can be understood without the algorithm(s) being explicitly programmed to make such predictions or decisions. In particular, decision tree-based machine learning models are machine learning models.

The terms “comprises,” “includes,” “includes,” “has,” “has,” “with,” or any other variation thereof, as may be used herein, are intended to cover non-exclusive inclusion. For example, a method or device that includes or has a list of elements is not necessarily limited to those elements, but may include other elements that are not expressly listed or that are inherent to such method or device.

Furthermore, unless expressly stated to the contrary, “or” refers to an inclusive or and not an exclusive “or”. For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

The terms “a” or “an” as used herein are defined to mean “one or more”. The terms “another” and “another” as well as any other variant thereof are to be understood in the sense of “at least one further”.

The term “plural” or “multiple” as used herein is to be understood in the sense of “two or more”.

The terms “configured” or “set up” to fulfill a specific function (and respective modifications thereof) as may be used herein are understood to mean, in the sense of the invention, that the corresponding device is already in a configuration or setting in which it fulfills the function can execute or at least it can be set - i.e. configurable - so that it can carry out the function after the appropriate setting. The configuration can be carried out, for example, by appropriately setting parameters of a process flow or switches or the like to activate or deactivate functionalities or settings. In particular, the device can have several predetermined configurations or operating modes, so that the configuration can be carried out by selecting one of these configurations or operating modes.

Various exemplary embodiments of the method are described below, each of which, unless this is expressly excluded or is technically impossible, can be combined in any way with each other and with the other aspects of the present solution described.

In some embodiments, the anchor points are points from the radar point cloud associated with the measurement data. This simplifies the determination of the anchor points, since no new points have to be created in the spatial area, but a selection can be made from the existing points in the radar point cloud.

In some embodiments, the anchor points are determined such that each of the anchor points within the spatial region is at a greatest distance from the respective remaining anchor points. This allows the transformations to be distributed across the spatial area in the best possible way. This also allows different types of transformations to be carried out, especially when compared to anchor points that are close together. The anchor points can be determined in such a way that a first anchor point is determined randomly or alternatively also determined manually. A second anchor point is then selected with the greatest possible distance from the first anchor point. All subsequent anchor points are then determined with the greatest possible distance from the anchor points that have already been determined. The so-called “Farthest Point Sampling” (FPS) can be used, in particular to select the anchor points from the set of radar points. Alternatively, all anchor points could be determined randomly, but then the greatest possible distance between the anchor points is no longer guaranteed.

In some embodiments, each of the transformations includes at least one of rotation, scaling, and translation with respect to the respective anchor point. A transformation can include any of the above or a combination thereof. The values of rotation, scaling and translation can be determined randomly. These transformations can be easily applied to the points of the radar point cloud, for example using appropriate matrices. Optimized value ranges of the transformations can be determined using hyperparameter optimization, e.g. Bayesian optimization.

gewählt werden, wobei d der Betrag des Abstands von einem Punkt zum Ankerpunkt ist. Der Wert σ kann beispielsweise 0,5 betragen.In some embodiments, each of the transformations is weighted point by point with a weighting factor, the weighting factor depending on a distance between a point of the radar point cloud to be transformed and the respective anchor point. By weighting lets the effect of the respective transformation on the individual points of the point cloud can be influenced. “Pointwise” means that each point in the point cloud receives a weighting factor. The weighting factors are advantageously chosen so that the transformation has a smaller effect on points that are at a greater distance from the corresponding anchor point than on points that are closer to the anchor point. To do this, a transformed point can be multiplied by the weighting factor or, if the weighting factors are chosen accordingly, divided. In this way, local transformations arise, so to speak, with the respective anchor point as the center. For example, the weighting factors can be selected according to a Gaussian distribution depending on the distance to the anchor point. In other words, this results in “smoothing” with a Gaussian kernel. The weighting factor w can be, for example, as $w=e^{-\frac{d^2}{{\sigma }^2}}$

can be chosen, where d is the amount of the distance from a point to the anchor point. The value σ can be 0.5, for example.

In some embodiments, the transformed point clouds assigned to the anchor points are combined by forming a point-by-point average of the transformed point clouds assigned to the anchor points. In this way, an augmented point cloud can be easily generated from the transformed point clouds. Several transformed points are assigned to each point of the radar point cloud, whereby the number corresponds to the number of anchor points, since a transformed point cloud is assigned to each anchor point.

In some embodiments, the training data is generated by a weighted sum of the measurement data and the augmented measurement data. By linking the augmented measurement data obtained with the actual measurement data, it can be ensured that the training data has a relationship to the actual measurement data and, for example, does not deviate too arbitrarily from the measurement data. Weighting can be done with a parameter, for example a difficulty parameter α, which can take values from 0 to 1. Here the augmented measurement data can be weighted with α and the actual measurement data with (1 - α). The difficulty parameter can also be determined through appropriate hyperparameter optimization.

The seating arrangement may have a plurality of seats, wherein the seat occupancy state may be an individual or cumulative seat occupancy state of the seats.

In some embodiments, the set of radar points for the radar point cloud is divided into several clusters, each containing a subset of the radar points, by means of cluster formation depending on the respective spatial position of the radar points in relation to the seats, in order to assign one of the clusters that is spatially closest to each of the seats assigned individually, with the augmented measurement data being generated individually for each cluster. The seat occupancy state of the seat arrangement is determined for each of the seats as a function of the radar point cloud determined for the associated cluster in order to obtain an evaluation result, in particular a classification result, which characterizes a seat occupancy state of the respective seat. The information to be output is then defined depending on the individual evaluation results for the different seats. If the augmented measurement data is generated individually for each cluster, the efficiency of the training data can be increased because the entire radar point cloud does not have to be used for the training data. Alternatively, it is also possible to apply the augmentation to the entire radar point cloud.

In some embodiments, the radar point cloud is segmented into several clusters by assigning a subset of the radar points of the respective radar point cloud to each of the seats as a cluster depending on their respective position in such a way that the radar points of the cluster are in a defined closed, in particular cuboid, spatial area in the environment of the seat. This enables a particularly simple and less computationally intensive cluster formation and thus seat-related seat occupancy recognition, whereby the location (position and orientation) and the shape of the spatial area is or can be defined in such a way that it corresponds to that of a typical object to be recognized, in particular a person, on a The seat of the seating arrangement generally overlaps the area of space occupied.

In some embodiments, for the augmented point cloud, the set of its points is divided into several clusters, each containing a subset of the points, by means of cluster formation depending on the respective spatial position of the points in relation to the seats, in order to assign one to each of the seats that is spatially closest to it to assign the clusters individually. In other words, clustering occurs (also) after the augmentation of the radar point cloud. This can further improve the result as some points might lie outside an original cluster after the augmentation, i.e. after the transformations. Such points are no longer included in the clustering after the augmentation attributed to the original cluster. In particular, as described above, a cluster can be created using corresponding spatial areas assigned to the individual seats, which are preferably cuboid and essentially cover the area in which a person is expected to be at the respective seat.

In some embodiments, the individual radar points of the radar point cloud are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: (i) a Doppler shift value of the radar signal to the respective radar point; (iii) a signal-to-noise ratio value of the radar signal to the respective radar point. These parameters can be used in particular for pre-filtering the radar point cloud as part of pre-processing prior to feature determination.

A second aspect of the solution presented here relates to a, in particular computer-implemented, method for the automated recognition of a, in particular seat-related, seat occupancy state of a seat arrangement with at least one seat (or equivalently: seat), in particular a seat in or for a vehicle, such as an automobile ( e.g. truck, car or bus). The method comprises: (i) acquiring, in particular receiving or generating, measurement data that represents an associated radar point cloud. Each radar point cloud is or was obtained on the basis of a radar scan of a spatial area that at least partially surrounds the seating arrangement; (ii) determining, in particular estimating, a seat occupancy state of the seat arrangement using an evaluation model which, depending on the radar point cloud, delivers one of several predefined possible seat occupancy states of the seat arrangement as an evaluation result; and (iii) outputting information defined depending on the evaluation result. The method according to the second aspect uses an evaluation model, in particular a machine learning model, which was trained using the method according to the first aspect.

Using the method according to the second aspect, an evaluation result (in particular in the sense of a prediction or classification) that characterizes the seat occupancy state of the seat arrangement can be obtained on the basis of a radar point cloud, which is obtained by means of a radar scan of a spatial area surrounding the seat arrangement. In particular in the vehicle context (especially for automobiles), radar-based solutions can be implemented that can reliably detect seat occupancy status (in particular exclusively) on a radar basis and, on the basis of this, certain functionalities or systems, such as a seat belt warning or an airbag system, overall or selectively activate, deactivate or control/regulate. By training with the augmented measurement data, the quality of the evaluation result when actually detecting a seat occupancy status can be improved, since a large amount of data can be used for training.

The information to be output can in particular represent the evaluation result itself. It can also be a detectable signal, in particular one that can be detected with a human sense, such as a warning, or a control signal for controlling a signal source or a data signal carrying the information.

Various exemplary embodiments of the method according to the second aspect are described below, each of which, unless this is expressly excluded or is technically impossible, can be combined in any way with each other and with the other described other aspects of the present solution. Embodiments of the method according to the first aspect may also be applicable to the method according to the second aspect and vice versa. In other words, features of the method that are used when training the evaluation model can equally be used to detect a seat occupancy state, and vice versa.

In some embodiments, the seating arrangement has a plurality of seats for which a seat occupancy status is to be determined individually or cumulatively as part of the method. For each (individual) radar point cloud, the set of its radar points is divided into several clusters, each containing a subset of the radar points, by means of cluster formation depending on the respective spatial position of the radar points in relation to the seats, in order to assign one of the clusters that is spatially closest to each of the seats assign individually. The seat occupancy state of the seat arrangement is determined for each of the seats as a function of the radar point cloud determined for the associated cluster in order to obtain an evaluation result, in particular a classification result, which characterizes a seat occupancy state of the respective seat. The information to be output is then defined depending on the individual evaluation results for the different seats.

In some related embodiments, each radar point cloud is segmented into multiple clusters by assigning each of the seats as a cluster Subset of the radar points of the respective radar point cloud is assigned depending on their respective position so that the radar points of the cluster lie in a defined closed spatial area, in particular a cuboid, in the vicinity of the seat. This enables a particularly simple and less computationally intensive cluster formation and thus seat-related seat occupancy recognition, whereby the location (position and orientation) and the shape of the spatial area is or can be defined in such a way that it corresponds to that of a typical object to be recognized, in particular a person, on a The seat of the seating arrangement generally overlaps the area of space occupied.

In the case of a multi-seat seating arrangement, seat-related, i.e. seat occupancy recognition is made possible for each seat, which is particularly advantageous or even necessary if a seat-related reaction is to be made to the recognized seat occupancy, for example for a specific seat depending on its recognized seat occupancy a specific functionality or a specific system, such as a seat-related airbag system, a seat-related seat belt warning or a seat-related seat heating, is to be activated or deactivated or otherwise controlled. In some embodiments, the cluster formation can in particular take place in such a way that the clusters are disjoint, so that no radar point is assigned to two different clusters.

In some embodiments, the or each individual radar point cloud is segmented into several clusters by assigning a subset of the radar points to each of the seats as a cluster depending on their respective position, in particular uniquely for each radar point, so that the radar points of the cluster are in a defined, closed, especially cuboid, room area in the vicinity of the seat. The assignment can in particular be carried out in such a way that each radar point is assigned to the cluster of the seat closest to it. In this way, the radar point cloud can be divided into clusters, i.e. subsets of the radar point cloud located in the vicinity of the respective seats, so that the determination of seat-specific seat occupancy states can be carried out specifically and therefore with high reliability on the basis of the cluster assigned to the respective seat.

In some embodiments, the method further comprises: determining the respective value of at least one defined parameter for characterizing radar point clouds. The evaluation model is or is defined in such a way that when determining the seat occupancy status of the seat arrangement, the evaluation result is determined depending on the respective values of the at least one parameter for the radar point clouds. In this way, the evaluation model can be determined and applied in a simplified manner, since instead of entire radar point clouds, only the values of at least one parameter have to be taken into account as input variables. In particular, the time course of the values can reflect a movement or a specific movement pattern of one or more objects on the seat arrangement, so that the evaluation model can determine the seat occupancy status of the seat arrangement in a particularly reliable manner on this basis.

In some embodiments, for at least one of the parameters, the series of their respective values is analyzed to determine whether a periodic course, in particular a course corresponding to a periodic breathing pattern, of the parameter is detected. The evaluation result is then determined depending on the result of the analysis. In particular, not only can the presence of any object on a seat of the seating arrangement be detected, but a distinction can also be made with high reliability between living objects present on the seating arrangement, especially people and mammals such as pets (e.g. dogs). In particular, it is possible to define the information to be output according to the method depending on the recognition or non-recognition of such a periodic course. In particular, one or more detected frequencies of the periodic course can also be taken into account, in particular in such a way that the information is defined depending on whether the or a frequency lies within a specific frequency range, such as a typical respiratory frequency range. In particular, a seat belt warning functionality or an airbag system can be controlled depending on whether a breathing frequency and thus, with high probability, a person has been detected on the seat arrangement or a specific seat thereof.

In some embodiments, the or one of the parameters is or is determined by the number of radar points in the radar point cloud or a variable dependent thereon. If, during radar scanning, the number of radar points in the radar point cloud generated depends on the extent to which the scanned object moves, the extent of the movement can be reflected in the parameter(s), in particular with regard to the aforementioned detection of a breathing frequency of the object . In addition to the number of radar points, other characteristics of the point cloud (or a cluster), such as a position, can also be used as a parameter the center of gravity of the point cloud (or a cluster), the density of the points (of a cluster) and/or an average Doppler value.

In some embodiments, the evaluation model includes a trained machine learning model. Data that represents at least one radar point cloud or values of one or more specific parameters are made available to the machine learning model as input data in order to receive the evaluation result as its output. This allows a particularly flexible and adaptable implementation of the evaluation model, whereby machine learning can be used to continuously improve the evaluation model and thus the quality and reliability of the seat occupancy detection. The evaluation result can in particular indicate a class of a seat occupancy status classification. The machine learning model can in particular be a decision tree-based model or a model based on an artificial neural network. As explained above in connection with the method according to the first aspect, the evaluation model is trained with training data before use in the method according to the second aspect, augmented data being used for this purpose in order to increase the amount of data provided to the evaluation model for the Training can be made available as input data.

In some embodiments, outputting the information includes driving a signal source depending on the information to cause the signal source to output a defined signal depending on the driving. The signal source can in particular be an audio source, an optical signal source, in particular a display device for images or text, and/or a haptic actuator or a combination of at least two of the aforementioned signal sources. The detected seat occupancy status can be communicated to a user based on the signaling or used to control another technical system, such as an airbag system.

In some of these embodiments, the signal source is controlled as a function of the information such that it outputs a signal, in particular defined by the control, when the information results from an evaluation result, according to which at least one seat of the seating arrangement is occupied and/or a selected predetermined one seat occupancy status exists.

In some embodiments, the method further comprises: (i) detecting a seat belt fastening state of at least one seat of the seating assembly or receiving seat belt information identifying this seat belt fastening state; (ii) the signal source is controlled as a function of the seat belt information and the information from the evaluation result in such a way that it outputs a belt fastening warning signal if, according to the information, at least one seat of the seat arrangement is occupied and/or a selected predetermined seat occupancy state is present and indicates seat belt information, that the associated seat belt of the seat is not fastened. In this way, a radar-based seat belt fastening testing and warning system can be achieved, particularly with regard to detection only.

In some embodiments, the individual radar points of the radar point cloud are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: (i) a Doppler shift value of the radar signal to the respective radar point; (iii) a signal-to-noise ratio value of the radar signal to the respective radar point. These parameters can be used in particular for pre-filtering the radar point cloud as part of pre-processing prior to feature determination.

In particular, in some of these embodiments, the determination of the seat occupancy status of the seat arrangement based on the evaluation model can be carried out exclusively or at least predominantly on the basis of those radar points whose Doppler shift value is at or above a predetermined shift threshold that is different from zero. So only or at least predominantly so-called dynamic radar points are used as a basis for the evaluation, i.e. those radar points that indicate a movement of the scanned object whose Doppler shift value is at or above the shift threshold. This can be used in particular to further increase the quality, in particular the reliability, of the method, because static, i.e. essentially stationary points on the object surfaces, such as points on a seat of a seat, are not included in the evaluation or are only included in a smaller number than such Points that show dynamics and can therefore most likely be assigned to a living being, especially a person or an animal. This allows the utilization of the information obtained through the evaluation (e.g. for airbag control or a seat belt warning system) to take place in particular depending on whether a living being was detected or a static object surface.

A third aspect of the present solution relates to a system, in particular a data processing device, for automatically recognizing a, in particular respective, seat occupancy state of a seat arrangement with at least one seat, in particular with at least one vehicle seat in or for a vehicle. The system has a data processing device that is configured, in particular by means of a corresponding computer program, to carry out the method according to the second aspect to detect the seat occupancy status.

A fourth aspect of the present solution relates to a computer program or computer program product comprising instructions which, when executed on the data processing device of the system according to the third aspect, cause the system to carry out the method according to the second aspect.

The computer program can in particular be stored on a non-volatile data carrier. This is preferably a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program as such is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can be present as a file on a data processing unit, in particular on a server, and can be downloaded via a data connection, for example the Internet or a dedicated data connection, such as a proprietary or local network. In addition, the computer program can have a plurality of interacting individual program modules. The modules can in particular be configured or at least can be used in such a way that they can be executed on different devices (computers or processor units) in the sense of distributed computing, which are geographically spaced apart from one another and connected to one another via a data network.

The system according to the third aspect can accordingly have a program memory in which the computer program is stored. Alternatively, the system can also be set up to access a computer program available externally, for example on one or more servers or other data processing units, via a communication connection, in particular in order to exchange data with it that are used during the course of the method or computer program or outputs of the computer program represent.

A fifth aspect of the present solution relates to a vehicle comprising: (i) a seating arrangement with at least one seat; (ii) a radar sensor for radar scanning of the seat arrangement at least in sections; and (iii) a system according to the third aspect for automatically detecting a, in particular respective, seat occupancy state of the seat arrangement as a function of an at least section-wise radar scanning of the seat arrangement carried out by the radar sensor, in particular according to a method according to the second aspect.

The features and advantages explained in relation to the first and second aspects of the present solution also apply accordingly to the other aspects of the solution.

Further advantages, features and possible applications of the present solution result from the following detailed description in connection with the drawings.

• 1 schematisch eine beispielhafte Ausführungsform eines Fahrzeugs, das mit einem System zum automatisierten Erkennen eines Sitzbelegungszustands einer Sitzanordnung im Fahrzeug ausgerüstet ist;
• 2 schematisch das Fahrzeug aus 1, wobei hier der Beifahrersitz belegt ist;
• 3A eine beispielhafte zweidimensionale Darstellung einer durch einen Radarsensor des Fahrzeugs aus 2 aufgenommenen Radarpunktwolke;
• 3B eine beispielhafte Darstellung einer Clusterung der Radarpunktwolke aus 3A gemäß den Positionen der einzelnen Sitzplätze der Sitzanordnung;
• 4 ein Flussdiagramm zur Veranschaulichung einer beispielhaften Ausführungsform eines Verfahrens zum automatisierten Erkennen eines Sitzbelegungszustands einer Sitzanordnung;
• 5 ein Flussdiagramm zur Veranschaulichung einer beispielhaften Ausführungsform eines Verfahrens zum Trainieren eines Auswertungsmodells für das automatisierte Erkennen eines Sitzbelegungszustands einer Sitzanordnung;
• 6 eine beispielhafte Darstellung von vier Radarpunkten während der Augmentation; und
• 7 eine beispielhafte dreidimensionale Darstellung einer Radarpunktwolke mit einer daraus erhaltenen augmentierten Punktwolke.

This shows:

• 1 schematically an exemplary embodiment of a vehicle that is equipped with a system for automatically detecting a seat occupancy status of a seat arrangement in the vehicle;
• 2 schematically the vehicle 1 , whereby the passenger seat is occupied here;
• 3A an exemplary two-dimensional representation of a radar sensor of the vehicle 2 recorded radar point cloud;
• 3B an exemplary representation of a clustering of the radar point cloud 3A according to the positions of each seat in the seating arrangement;
• 4 a flowchart illustrating an exemplary embodiment of a method for automatically detecting a seat occupancy state of a seating arrangement;
• 5 a flowchart to illustrate an exemplary embodiment of a method for training an evaluation model for the automated recognition of a seat occupancy status of a seating arrangement;
• 6 an exemplary representation of four radar points during augmentation; and
• 7 an exemplary three-dimensional representation of a radar point cloud with an augmented point cloud obtained from it.

In the figures, the same reference numerals denote the same, similar or corresponding elements. Elements shown in the figures are not necessarily drawn to scale. Rather, the various elements shown in the figures are reproduced in such a way that their function and general purpose are understandable to those skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings, unless expressly stated otherwise. Functional units can in particular be implemented as hardware, software or a combination of hardware and software.

In the 1 Schematically illustrated exemplary embodiment of a vehicle 100 has a seat arrangement 105 with five individual seats or seats 105a to 105e. Each of the seats 105a to 105e is suitable for accommodating one person as a passenger of the vehicle 100. The vehicle 100 further includes a radar sensor 110 mounted within the vehicle cabin on the ceiling and configured to scan the seat assembly 105, at least substantially, using radar beams. Accordingly, the seats 105a to 105e, in particular their seating surfaces, are at least predominantly located within an observation field 110a that can be scanned by the radar sensor 110. In addition, the vehicle 100 has a system 115 for automatically detecting a seat occupancy state of the seat arrangement 105 depending on a radar scanning of the seat arrangement 105 carried out by the radar sensor 110, at least in sections with respect to the observation field 110a.

The system 115 in particular has a data processing unit 115a with at least one microprocessor and a memory 115b connected to the signal, in which a data processing unit 115a is used to carry out the following with reference to 4 described method for automatically recognizing a seat occupancy state of the seat arrangement 105 configured computer program is stored. Furthermore, the sensor data generated by the radar sensor 110 during the radar scanning or information already obtained from it through further processing can be or will be stored in the memory 115b.

This in 2 Vehicle 100 shown corresponds to the vehicle 1 , although here the passenger seat 105b is occupied by a person P. In the further discussion that follows 3A and 3B depends on the constellation 2 Referenced.

Below we will now turn to the 3A and 3B Reference is made, each representing a radar point cloud, whereby for the purpose of display, the respective, inherently three-dimensional radar point cloud was reduced to two dimensions by projecting the positions of the radar points of the radar point cloud onto a plane spanned by two of its dimensions.

In 3A is an exemplary radar point cloud 305 illustrated as captured as a result of a radar scan of the seating arrangement 105 by the radar sensor 110 during a specified time interval (measurement period). The location of the individual radar points within the radar point cloud 305 can be represented by spatial coordinates, for example Cartesian coordinates X and Y can be assigned to the drawing plane and correspondingly to each individual point. In reality, if the dimension reduction based on the drawing is ignored, there is a third coordinate Z for the third spatial dimension (cf. the one in 7 X, Y and Z coordinates shown).

If during radar scanning not only the spatial positions of the locations at which the radar beam is reflected by the scanned objects are recorded as coordinates, but also a respective Doppler shift is measured, then the individual radar points can be classified depending on the amount of this Doppler shift, in particular be divided into two different classes. The latter can be done, for example, by comparing the Doppler shift with a predefined shift threshold that corresponds to a specific shift speed. Depending on the result of the comparison, those radar points 310 which, according to the value of their associated Doppler shift, have no speed or a speed of the object surface at the reflection point that is below the displacement wave, can be classified as “static” radar points (in the 3A and 3B each shown with a filled black circle). Conversely, those radar points 315 that have a Doppler shift above the shift threshold can be classified as “dynamic” radar points 315 (in the 3A and 3B each shown with a black ring).

The classification of the radar points 310 and 315 according to their Doppler shift is not absolutely necessary, but it can be used to process the radar point cloud 305, in particular as part of a pre-processing that takes place before its evaluation, in particular to filter it depending on the classification. At For example, such filtering could be carried out in such a way that only dynamic radar points 315 are taken into account for the evaluation, for example in order to only detect moving objects.

In 3B is the same radar point cloud 305 as in 3A shown. In addition, however, cuboid (3D case) or, in the present 2D representation, rectangular, selected spatial areas 325a to 325e are shown, which are spatially assigned to the respective position of the individual seats 105a to 105e. The definition of these spatial areas 325a to 325e can now be used to cluster the radar point cloud 305, with each radar point 310 or 315, as far as possible, being assigned to the spatial area 325a to 325e in which it lies. All radar points that are not located in one of the spatial areas 325a to 325e can remain ignored. It can be seen in particular that the areas 320 with a particularly high radar point density are in the area of the passenger seat 105b, on which according to 2 the person P is located.

4 shows a flowchart to illustrate an exemplary embodiment 400 of a method for automatically detecting a seat occupancy status of a seating arrangement. The method can in particular be designed as a computer-implemented method. For this purpose, it can be stored in particular in the memory 115b of the system 115 as a computer program and can be executed on the data processing unit 115a.

In the method 400, a radar point cloud 305 is captured by receiving radar measurement data, in the present example from the radar sensor 110 of the vehicle 100, in a step 410 and further processing it to form one or more radar point clouds.

The radar point cloud 305 that is now present can then be clustered in a further process 420 by checking for each of its radar points whether it is within one of the defined spatial areas 325a to 325e (cf. 3B) lies and, if necessary, in which. Thus, each of the points can either be assigned to one of the spatial areas 325a to 325e or to the other observation field. All radar points that lie within the same spatial area 325a to 325e are combined into a respective cluster. As a result, each of the seats 105a to 105e is assigned a corresponding cluster of the radar point cloud 305. This forms the basis for an evaluation to be carried out individually for each seat 105a to 105e as to whether the respective seat 105a to 105e is or was occupied while the radar point cloud 305 was formed or not.

In order to facilitate the subsequent evaluation of the clustered radar point cloud 305, a corresponding parameter K can be determined for each of the clusters in a process 430, whereby this parameter K can be defined in particular as the number of radar points in the cluster. If no filtering according to the Doppler shift value has taken place, this can involve a joint count of both the static and the dynamic radar points 310 and 315, respectively. However, if the static radar points 310 were previously filtered out, it is only a matter of counting the dynamic radar points 315. Alternatively or additionally, the static radar points 310 can also be counted.

The parameter K can now be evaluated for the cluster for seat 105b (the same can be done analogously for the respective clusters for the other seats). For this purpose, in process 440 the parameter K (or a time course of the parameter K is made available to an evaluation model as an input variable. This can in particular be a model based on machine learning, such as an artificial neural network or a decision tree-based model . “Decision tree(s)”). The training and possibly validation data used for the previous training can be structured in such a way that they each have the parameter K for a large number of different radar point clouds or clusters thereof as well as for each course of K contain an assigned correct class of a classification of possible seat occupancy states. In this way, the model can be trained and validated in the sense of supervised learning. In the simplest case, seat occupancy states indicate whether the seat is occupied or not. However, there are also more developed ones Classifications are conceivable in which, in the case of the presence of an object, the respective class also indicates what type of object it is, for example a moving or a stationary one, and in the case of a moving object, in particular, whether it is a person (basically recognizable in particular by a breathing pattern over the course of the parameter K).

If in process 440 a seat occupancy state for the seating arrangement 105 has been determined based on the evaluation model, in particular for one or more of its seats 105a to 105e individually, this result can be output as corresponding information in process 445, for example on a user interface of the vehicle or in the form of Data for further processing Processing by one or more other systems, in particular vehicle systems.

In the present example, this information is intended to be used in particular to check whether a seat belt warning signal is to be issued or not depending on the seat occupancy status of a respective seat 105a to 105e and the result of a test as to whether a corresponding seat belt has been fastened for this seat or not.

For this purpose, in process 450 it can be checked whether the seat belt is fastened to the seat in question (here, for example, to seat 105b) and in step 455 a functionality of the vehicle 100 depending on the information about the seat occupancy status output in process 445 and the status determined in process 450 the seat belt can be controlled. In particular, this can be done in such a way that in process 455 a signal source for outputting a particularly optical and/or acoustic belt status signal is activated in order to possibly signal to one or more other occupants of the vehicle that a seat is occupied but the seat belt is not fastened there. The method then branches back to step 410 to start another loop run.

5 shows a flowchart to illustrate an exemplary embodiment 500 of a method for training an evaluation model, which can be used in the method 400 for automatically recognizing a seat occupancy status of a seating arrangement. The procedure 500 is performed simultaneously with reference to 6 explained using four exemplary points a, b, c and d.

First, in a step 510, measurement data is recorded which represents an assigned radar point cloud, such as the radar point cloud 305. In 6 To illustrate the process, four radar points a, b, c and d are shown at the top. Anchor points are then determined from the set of radar points. For example, a first anchor point can be determined randomly, and all other anchor points can be determined at a greatest distance from the previously determined anchor points. For example, 2, 3, 4 or 5 anchor points can be determined. The number of anchor points can vary depending on the application, although in one example four anchor points proved to be optimal. This can be determined through hyperparameter optimization, for example Bayesian optimization. In the in 6 In the example shown, point a is selected as the anchor point. All points a, b, c and d are then copied to get points a_a, b_a, c_a and d_a.

A random transformation of the points with respect to the anchor point a now takes place in step 530. This includes a randomly selected rotation, a randomly selected scale, and a randomly selected translation. So the points a_a_aug, b_a_aug, c_a_aug and d_a_aug are obtained. An extent of the respective transformation can also be determined through hyperparameter optimization. After the transformation, smoothing takes place, for example using a Gaussian kernel.

This ensures that points further away from the anchor point undergo a smaller transformation than points closer to the anchor point. Appropriate weights can be selected for this. This results in the points a_a_aug_smooth, b_a_aug_smooth, c_a_aug_smooth and d_a_aug_smooth.

The transformation and smoothing is repeated for each of the anchor points. For example, the points a_c_aug_smooth, b_c_aug_smooth, c_c_aug_smooth and d_c_aug_smooth would result for the anchor point c. In step 550, all transformed and smoothed point clouds are then combined. This can be done, for example, by forming a point-wise average, for example a_aug = (a_a_aug_smooth + a_c_aug_smooth)/2, etc., to obtain points a_aug, b_aug, c_aug and d_aug (not shown in 6 ).

The obtained augmented points a_aug, b_aug, c_aug and d_aug are now adjusted in step 560 using a difficulty parameter α. The obtained augmented points are coupled to the original points a, b, c and d so that the resulting augmented point cloud does not deviate too much from the original radar point cloud. A result can be obtained by a_result= a_aug * α + a * (1-α), same for b_result, c_result and d_result. In this way, a machine learning model can be obtained that, through training with the augmented measurement data, provides a high level of accuracy in recognizing a seat occupancy state.

Since the positions of the obtained points a_result, b_result, c_result and d_result may differ from the positions of the original points a, b, c and d, the augmented point cloud is clustered (step 570) in order to then obtain suitable input data for training the evaluation model (Step 580).

In 7 Finally, another example of a radar point cloud 705 is illustrated. The illustration shows the radar point cloud 705 as a three-dimensional point cloud with the coordinates X, Y and Z. The view corresponds to a seat occupancy like that in 2 Seat occupancy shown with one person P in the front passenger seat. This is in 7 can be recognized by the concentration of the radar points 715 at the top right (which corresponds to the front right of the vehicle). The majority of points 715 are therefore in a cluster that is assigned to the passenger seat. Also shows 7 an augmented point cloud 710 with the points 720, which is represented by open rings.

It can be seen that the augmented point cloud 710 deviates slightly from the actual radar point cloud 705. In particular, there are larger deviations within the cluster of the passenger seat, whereas the deviations in the edge areas are smaller (in the illustration, the points 720 of the augmented point cloud 710 are mostly covered by the points 715 of the radar point cloud 720 and are therefore in 7 not visible).

While at least one exemplary embodiment has been described above, it should be noted that a large number of variations exist. It should also be noted that the exemplary embodiments described are only non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide guidance to those skilled in the art for implementing at least one exemplary embodiment, with the understanding that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from that set forth in the appended claims the specified subject matter and its legal equivalents are deviated from.

REFERENCE SYMBOL LIST

P

Person in the passenger seat

100

vehicle

105

Seating arrangement

105a-e

Seats or seats

110

Radar sensor

110a

Observation field of the radar sensor 110

115

System for automated recognition of seat occupancy status

115a

Data processing unit

115b

Storage

305

Radar point cloud

310

static radar points

315

dynamic radar points

320

Areas of the radar point cloud 305 with high radar point density

325a-e

Spatial areas for cluster definition

400

Method for automatically recognizing a seat occupancy status

410-455

individual processes or procedural steps within the framework of procedure 400

500

Method for training an evaluation model

510-580

individual processes or procedural steps within the framework of procedure 500

705

Radar point cloud

710

augmented point cloud

715

Point of radar point cloud 705

720

Point of the augmented point cloud 710

Claims (17)

Method for training an evaluation model for an automated recognition of a seat occupancy state of a seat arrangement (105) with at least one seat (105a-e), the method comprising: - Acquiring measurement data which represent an associated radar point cloud (305), the radar point cloud (305 ) was or is obtained on the basis of a radar scan of a spatial area surrounding the seat arrangement (105) at least in sections and is assigned to one of several predefined possible seat occupancy states of the seat arrangement (105); - Generating augmented measurement data, which represent an associated augmented point cloud, from the measurement data associated with the radar point cloud (305), the generation of the augmented measurement data comprising: - determining multiple anchor points within the spatial area; - Determine a transformation with respect to each of the anchor points, each of the transformation is applied to at least some of the points of the radar point cloud (305) in order to obtain a transformed point cloud assigned to the respective anchor point; and - combining the transformed point clouds associated with the anchor points to obtain the augmented point cloud associated with the augmented measurement data; and - generating training data from the measurement data and the augmented measurement data, the training data being made available to the evaluation model as input data in order to obtain an evaluation result as its output, which is assigned to the seat occupancy status of the seat arrangement (105). Procedure according to Claim 1 , where the anchor points are points from the radar point cloud (305) assigned to the measurement data. Procedure according to Claim 1 or 2 , whereby the anchor points are determined in such a way that each of the anchor points within the spatial area has a greatest distance from the remaining anchor points. Method according to one of the preceding claims, wherein each of the transformations comprises at least one of rotation, scaling and translation with respect to the respective anchor point. Method according to one of the preceding claims, wherein each of the transformations is weighted point by point with a weighting factor, the weighting factor depending on a distance between a point of the radar point cloud to be transformed and the respective anchor point. Method according to one of the preceding claims, wherein the joining of the transformed point clouds assigned to the anchor points is carried out by forming a point-by-point average of the transformed point clouds assigned to the anchor points. Method according to one of the preceding claims, wherein the training data is generated by a weighted sum of the measurement data and the augmented measurement data. Method according to one of the preceding claims, wherein: the seating arrangement (105) has a plurality of seats (105a-e), the seat occupancy state being an individual or cumulative seat occupancy state of the seats (105a-e); and for the radar point cloud (305), the set of its radar points (310, 315) by means of cluster formation depending on the respective spatial position of the radar points (310, 315) in relation to the seats (105a-e) into several subsets of the radar points (310, 315) containing clusters is divided, in order to individually assign one of the clusters that is spatially closest to each of the seats (105a-e), the augmented measurement data being generated individually for each cluster. Procedure according to Claim 8 , wherein the radar point cloud (305) is segmented into several clusters by assigning a subset of the radar points (310, 315) of the respective radar point cloud (305) to each of the seats (105a-e) as a cluster in such a way that the radar points (310, 315) of the cluster are located in a defined, closed spatial area in the vicinity of the seat (105a-e). Method according to one of the preceding claims, wherein: the seating arrangement (105) has a plurality of seats (105a-e), the seat occupancy state being an individual or cumulative seat occupancy state of the seats (105a-e); and for the augmented point cloud (305), the set of its points (310, 315) by means of cluster formation depending on the respective spatial position of the points (310, 315) in relation to the seats (105a-e) into several subsets Clusters containing points (310, 315) are divided in order to individually assign one of the clusters that is spatially closest to each of the seats (105a-e). Method according to one of the preceding claims, wherein the individual radar points (310, 315) of each radar point cloud (305) are each represented by a position of the respective radar point in three-dimensional space and by at least one of the following parameters: - A Doppler shift value of the radar signal to the respective radar point; - A signal-to-noise distance value of the radar signal to the respective radar point. Method for automatically detecting a seat occupancy state of a seat arrangement (105) with at least one seat (105a-e), the method comprising: - Acquiring measurement data that represent an associated radar point cloud (305), each radar point cloud (305) being based on a radar scan a spatial area surrounding the seat arrangement (105) at least in sections has been or will be obtained; - Determining a seat occupancy status of the seat arrangement (105) using an evaluation model, which is according to a method according to one of Claims 1 until 11 was created and dependent ness from the radar point cloud provides one of several predefined possible seat occupancy states of the seat arrangement (105) as an evaluation result; and - outputting information defined depending on the evaluation result. Procedure according to Claim 12 , wherein outputting the information includes driving a signal source depending on the information to cause the signal source to output a defined signal depending on the driving, wherein the signal source is driven depending on the information so that it outputs a signal , if the information results from an evaluation result, according to which at least one seat of the seating arrangement (105) is occupied and / or a selected predetermined seat occupancy state is present. Procedure according to Claim 13 , further comprising: detecting a seat belt fastening state of at least one seat of the seat assembly (105) or receiving seat belt information identifying this seat belt fastening state; wherein the signal source is controlled as a function of the seat belt information and the information from the evaluation result in such a way that it outputs a belt fastening warning signal if, according to the information, at least one seat of the seat arrangement (105) is occupied and/or a selected predetermined seat occupancy state is present and indicates seat belt information, that the associated seat belt of the seat is not fastened. System (115) for automatically detecting a seat occupancy status of a seat arrangement (105) with at least one seat (105a-e), the system (115) having a data processing device which is configured to detect the seat occupancy status using the method according to one of Claims 12 until 14 to carry out. Computer program or computer program product, having instructions that, when executed, are carried out on the data processing device of the system (115). Claim 15 , cause the system (115) to carry out the procedure according to one of the Claims 12 until 14 to carry out. Vehicle (100), comprising: a seating arrangement (105) with at least one seat (105a-e); a radar sensor (110) for radar scanning of the seat arrangement (105) at least in sections; and a system (115). Claim 15 for automatically detecting a seat occupancy state of the seat arrangement (105) as a function of an at least sectional radar scanning of the seat arrangement (105) carried out by the radar sensor (110).

Images