EP4078433A1 - Procédé et dispositif pour générer et fournir une base de données dans laquelle sont stockées des pièces de données de capteur destinées à être utilisées dans le matelassage - Google Patents

Procédé et dispositif pour générer et fournir une base de données dans laquelle sont stockées des pièces de données de capteur destinées à être utilisées dans le matelassage

Info

Publication number
EP4078433A1
EP4078433A1 EP20829549.3A EP20829549A EP4078433A1 EP 4078433 A1 EP4078433 A1 EP 4078433A1 EP 20829549 A EP20829549 A EP 20829549A EP 4078433 A1 EP4078433 A1 EP 4078433A1
Authority
EP
European Patent Office
Prior art keywords
sensor data
database
data
patches
sensor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP20829549.3A
Other languages
German (de)
English (en)
Inventor
Peter Schlicht
Fabian HÜGER
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Volkswagen AG
Original Assignee
Volkswagen AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Volkswagen AG filed Critical Volkswagen AG
Publication of EP4078433A1 publication Critical patent/EP4078433A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G06F18/24133—Distances to prototypes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • the invention relates to a method and a device for generating and providing a database with sensor data patches stored therein for use in quilting.
  • the invention also relates to an assistance system for a vehicle as well as a computer program and a data carrier signal.
  • Machine learning for example based on neural networks, has great potential for use in modern driver assistance systems and automated vehicles.
  • Functions based on deep neural networks process sensor data (e.g. from cameras, radar or lidar sensors) in order to derive relevant information from it.
  • sensor data e.g. from cameras, radar or lidar sensors
  • This information includes, for example, a type and a position of objects in the surroundings of the motor vehicle, a behavior of the objects or a road geometry or topology.
  • CNN convolutional neural networks
  • CNN convolutional neural networks
  • CNN convolutional neural networks
  • input data e.g. image data
  • CNN convolutional neural networks
  • the convolution network independently develops feature maps based on filter channels that process the input data locally in order to derive local properties. These feature cards are then processed again by further filter channels, which derive more valuable feature cards from them.
  • the deep neural network On the basis of this information compressed from the input data, the deep neural network finally derives its decision and makes it available as output data.
  • a method and a device for recognizing anomalies in sensor data are known from US 2019/0135300 A1.
  • the method includes, for example, obtaining the first sensor data from a first sensor and the second sensor data from a second sensor, the first sensor of a first sensor type being different from a second sensor type of the second sensor; Generating first encoded sensor data based on the first sensor data and second encoded sensor data based on the second sensor data; Generating a contextual fused sensor data representation of the first and second sensor data based on the first and second encoded sensor data; Generating the first and second reconstructed sensor data based on the contextual representation of the merged sensor data; Determining a deviation estimate based on the first and second reconstructed sensor data, the deviation estimate being representative of a deviation between: (a) the first reconstructed sensor data and (b) the first sensor data; and detecting an anomaly in the deviation estimate, the anomaly indicating an error associated with the first sensor.
  • An anomaly can be detected, for example, by means of an auto-
  • the invention is based on the object of creating a method and a device for generating and providing a database with sensor data patches stored therein for use in quilting, with which sensor data can be improved and robustized against adversarial attacks.
  • a method for generating and providing a database with sensor data patches stored therein for use in quilting, with several output domain databases being made available and / or being accessed, with sensor data being stored in each of the several output domain databases, which are stored by means of sensors that are calibrated to one another were recorded, with sensor data from the A plurality of output domain databases are obtained, a number parameter and a size parameter being obtained, a number of sensor data patches determined by the number parameter being generated from the received sensor data, each with a size specified by the size parameter being generated and stored in the database, the number being based on the multiple Initial domain databases is split, and the generated database is provided.
  • a device for generating and providing a database with stored sensor data patches for use in quilting comprising a data processing device, the data processing device being set up to provide and / or access multiple output domain databases, sensor data being stored in each of the multiple output domain databases which were recorded by sensors calibrated to one another, furthermore to obtain sensor data from the multiple output domain databases, to obtain a number parameter and a size parameter, and to generate a number of sensor data patches specified by the number parameter, each with a size specified by the size parameter, from the sensor data obtained and to be stored in the database, and in this case to divide the number proportionally to the several output data domain databases, and to have the database generated ready ll.
  • the method and the device make it possible to generate and provide a database as a starting point for quilting sensor data.
  • the database can be provided in a data domain-agnostic manner, since several output data domains or associated output domain databases are used from which sensor data for generating sensor data patches are obtained, in particular received.
  • Sensor data patches are generated from the sensor data obtained and stored in the database.
  • a sensor data patch can also be referred to as a data block.
  • the sensor data patches in particular form subsymbolic subsets of the sensor data.
  • the sensor data patches have a size that is determined by the size parameter.
  • the size defines an image section or a number of picture elements that a sensor data patch should contain, for example image sections with a size of 8x8 picture elements each.
  • the number parameter defines a number (e.g. 10000, 100000 etc.) of sensor data patches in the database.
  • sensor data are proportionally used from all output domain databases in order to create the sensor data patches.
  • the generated database is then made available. The provided The database can then be used in a quilting step to replace or reconstruct sensor data of the same type from a plurality of sensors piece by piece and thereby to robustize them against adversarial disturbances.
  • One advantage of the method and the device is that a database can be created for use in quilting sensor data that are assigned to different data domains.
  • a database can be created for use in quilting sensor data that are assigned to different data domains.
  • the method and the device are used in particular to provide a database for a quilting method for robustizing sensor data against adversarial disturbances.
  • the provision of the starting domain databases can include receiving and / or recording the sensor data contained therein by means of an interface and / or by means of sensors and generating the starting domain databases from the received and / or recorded sensor data.
  • the output domain databases include, in particular, sensor data from the same sensors or sensor combinations, for example, in each case sensor data from a sensor combination of a camera and a lidar sensor.
  • the provision of the generated database can include loading the generated database into a storage device of a robustification device for robustizing sensor data against adversarial disturbances.
  • a data packet comprising the database generated is transmitted to the robustification device and loaded there into the storage device.
  • a vehicle is in particular a motor vehicle.
  • a vehicle can also be another land, rail, water, air or space vehicle, for example a drone or an air taxi.
  • the method and the device can also be used in other areas of application, for example in industrial production or in robotics.
  • Quilting refers in particular to the piece-wise replacement of sensor data, which can also be referred to as piece-wise reconstruction of the sensor data.
  • image quilting is also used in connection with image data.
  • a lot of replaced Sensor data forms a reconstruction data domain or is included in a reconstruction data domain. If, for example, images from a camera are involved, the camera image is divided into several partial sections. For this purpose, small, rectangular image sections (also known as patches) can be defined.
  • the individual partial or image sections are compared with partial sections, hereinafter referred to as sensor data patches, which are stored in the database. The comparison takes place on the basis of a distance measure which is defined, for example, via a Euclidean distance on picture element vectors.
  • a partial or image section is linearized as a vector.
  • a distance is then determined using a vector standard, for example using the L2 standard.
  • the partial or image excerpts from the recorded sensor data are each replaced by the closest or most similar sensor data patch from the database. It can be provided here that a minimum distance must be maintained or that at least no identity may exist between the partial section from the sensor data and the sensor data patch. If the sensor data have a different form (eg lidar data) or a different format, the piece-by-piece replacement takes place in an analogous manner. The piece-by-piece replacement takes place for all partial sections of the recorded sensor data, so that replaced or reconstructed sensor data are then available. After the piece-wise replacement, that is to say after the quilting, an effect of the adversarial disturbances in the replaced or reconstructed sensor data is eliminated or at least reduced.
  • the database generated by means of the method described in this disclosure is used in a method for robustizing sensor data against adversarial disturbances, with sensor data being obtained from at least one sensor, the sensor data obtained from the at least one sensor in each case by means of quilting based on the generated in the Sensor data patches stored in the database are replaced piece by piece, the piece-by-piece replacement being carried out in particular in such a way that each replaced sensor data from different sensors (if any) are plausible to one another, and the piece-wise replaced sensor data are output.
  • the method for robustification is carried out in particular by means of an associated robustification device.
  • a robustification device is used for this purpose for robustizing sensor data against adversarial disturbances, comprising a computing device and a storage device, the computing device being set up to receive sensor data from at least one sensor, the sensor data obtained from the at least one sensor each by quilting on the basis of the in to replace sensor data patches stored in the database generated piece by piece, and the piece by piece replacement in particular in this way perform that each replaced sensor data of different sensors (if available) are plausible to each other, and output the piece-wise replaced sensor data.
  • a “plausibility” of replaced sensor data is intended to mean in particular that the replaced sensor data are physically plausible to one another.
  • a probability that the respectively replaced sensor data in the respective cross-sensor combination would also occur under real conditions, i.e. in the real world, should be as large as possible (in the sense of e.g. a maximum likelihood).
  • the replaced sensor data of the at least two sensors should be selected in such a way that the probability that these sensor data would also actually occur in this combination is maximized.
  • a plausibility between the respectively replaced sensor data means that a viewed image section in the replaced camera data and a locally and temporally corresponding sub-section from the replaced lidar data are selected in such a way that the Sensor data are consistent with one another, i.e. they are physically free of contradictions to one another.
  • the at least two sensors are a camera and a lidar sensor
  • the partial sections of the sensor data are each replaced in such a way that each replaced image section corresponds to a replaced partial section of the lidar data, as this is very likely also when simultaneously capturing Sensor data of the camera and the lidar sensor would result.
  • the sensor data or the sensors with which the sensor data stored in the output domain databases were recorded are, in particular, calibrated with respect to one another in terms of location and time, so that the sensor data of the sensors correspond to one another in terms of location and time or have common reference points in terms of time and location.
  • the sensor data originate in particular from sensors of different types, for example from a camera and a lidar or radar sensor.
  • the sensor data are physically plausible to one another, that is, the sensor data do not physically contradict one another.
  • the respective sensors or sensor data are selected as a function of a planned application scenario for the database generated.
  • the database is also generated on the basis of recorded sensor data from a camera (and a lidar sensor).
  • the sensor data recorded or stored in the initial domain databases are, in particular, definitely free from or less heavily burdened with potential adverse disturbances.
  • the sensor data can in principle be one-dimensional or multidimensional, in particular two-dimensional.
  • the sensor data can be two-dimensional camera images from a camera and two-dimensional or three-dimensional lidar data from a lidar sensor.
  • the sensor data patches can also be referred to as data blocks.
  • the sensor data patches in particular form subsymbolic subsets of the sensor data recorded or stored in the initial domain databases.
  • a data domain should in particular denote a total of sensor data which correspond to a specific context or whose data are similar in at least one property with regard to their origin.
  • a context can be, for example, a geographical context, e.g. a data domain can include data from one city, whereas a different data domain comprises data from another city, etc.
  • Such a context can also be a data creation paradigm (e.g. real-world sensor impressions, simulation data, synthetic or synthetically modified sensor data ).
  • the output data domains that are mapped by the output domain databases are, in particular, data domains in the context of automated driving.
  • Such can in particular be recorded sensor data (measurement data) from different application scenarios of an automated vehicle with and without basic truth, homologation data and / or simulation data as well as sensor data recorded in a vehicle fleet (which, for example, also include or depict rare events and / or typical error situations).
  • the sensor data recorded or stored in the output data domain databases are those sensor data that are recorded for a function for automated or partially automated driving of a vehicle and / or for driver assistance of the vehicle and / or for environment detection or for environment perception were.
  • a vehicle is in particular a motor vehicle. In principle, however, the vehicle can also be another land, air, water, rail or space vehicle, for example a drone or an air taxi.
  • An adversarial perturbation is, in particular, a deliberately made disruption of the input data of a neural network, for example provided in the form of sensor data, in which semantic content in the input data is not changed, but the disruption leads to the neuronal Net a wrong one Result inferred, that is, for example, a misclassification or an incorrect semantic segmentation of the input data undertakes.
  • a neural network is in particular a deep neural network, in particular a convolutional neural network (CNN).
  • the neural network is or is trained, for example, on a function for automated or partially automated driving of a vehicle and / or for perception of the surroundings, for example on the perception of pedestrians or other objects in captured camera images.
  • the method can be carried out as a computer-implemented method.
  • the method can be carried out by means of a data processing device.
  • the data processing device comprises in particular at least one computing device and at least one storage device.
  • a computer program is also created, comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method steps of the disclosed method in accordance with any of the described embodiments.
  • a data carrier signal is also created that transmits the aforementioned computer program.
  • Parts of the device in particular the data processing device, can be designed individually or collectively as a combination of hardware and software, for example as program code that is executed on a microcontroller or microprocessor.
  • the selection of a partial section from the sensor data in order to generate a sensor data patch from the selected partial section can take place in different ways.
  • the sensor data patches are generated randomly, that is to say for example by means of a Monte Carlo method, from the sensor data obtained.
  • the provision of the database generated includes loading the database generated into a storage device of a robustification device of an assistance system of at least one vehicle, wherein the robustification device is set up to replace acquired sensor data of at least one sensor piece by piece with the aid of the sensor data patches stored in the database by means of quilting, so that the replaced sensor data are robustized against adversarial disturbances.
  • the database is created and made available, for example, by means of a central server.
  • the database generated is then transmitted to a robustification device of an assistance system of at least one vehicle, for example via appropriately designed wired or wireless communication interfaces, and loaded into a storage device of the robustification device.
  • the replaced sensor data are supplied to a function for the automated or partially automated driving of the vehicle and / or for a perception of the surroundings.
  • the function can work more reliably based on the replaced sensor data.
  • the function Based on the supplied, replaced sensor data, the function generates at least one control signal and / or evaluation signal and makes this available.
  • the sensor data in the multiple output domain databases are at least partially marked or marked with at least one property information item, with at least one property parameter being obtained, and with the generation of the sensor data patches for the database taking into account the received at least one property parameter and the at least one item of property information is carried out.
  • the sensor data patches generated are marked with property information as a function of the at least one property parameter.
  • the at least one property parameter specifies in particular which property information is used or is provided for marking a sensor data patch, in particular in the form of at least one piece of patch property information assigned to it in the database.
  • Property information or patch property information can also be referred to as a “tag”.
  • the sensor data patches can be specifically marked with (patch) property information, as a result of which a search in the database generated can be accelerated during an application phase when replacing piece by piece by means of quilting.
  • Property information can include, for example, sensor properties (resolution, alignment, sensor model, etc.) of the sensors by means of which the sensor data were recorded.
  • Property information can include, for example, a point in time or time segment and / or a local desertification within the time segment in the sensor data, for example a typical pixel position (e.g. in x, y coordinates).
  • the typical pixel position can be calculated in the sensor data using statistical methods, for example. In this way, localization priorities can be created which can accelerate a search for suitable sensor data patches in a subsequent use of the database.
  • property information can also include a point in time (time of day, day of the week, month, season, etc.) and / or a location (e.g. a geographical position) at which or at which the sensor data was recorded.
  • a point in time time of day, day of the week, month, season, etc.
  • a location e.g. a geographical position
  • Property information can also describe the respective sensor data in more detail in terms of content, for example in that the property information includes a ground truth about the sensor data or a part thereof and / or context information about the sensor data or a part thereof.
  • a basic truth describes in particular which objects are mapped in the sensor data and / or how the sensor data are semantically segmented. In this way, semantic priorities can be provided which can accelerate a search for suitable sensor data patches in a subsequent use of the database.
  • Context information describes a context in which the sensor data was recorded (e.g. a weather, a time of day, a weekday, a month, a season, a season, a traffic situation, such as motorway, country road, city traffic, etc., road users, critical situations, etc. .). This allows context priorities to be created which can accelerate a search for suitable sensor data patches in a subsequent use of the database.
  • Property information can also be generated or provided on the basis of vehicle data; for example, vehicle data can be collected while the sensor data is being captured, with which the captured sensor data are then marked or from which property information is derived. Vehicle data can be queried and / or received via a controller area network (CAN) bus, for example become.
  • CAN controller area network
  • Property information can also include typical properties of sensor data or a partial section from sensor data of a sensor used in parallel or at the same time.
  • a sensor data patch for a camera image of a camera can be marked as property information with a sensor data patch of a lidar sensor that is typical for this. This allows geometric priors to be provided so that when a suitable sensor data patch is found in the database to replace a partial section of a camera image using the property information assigned by way of the marking in the database, an associated sensor data patch for the lidar sensor can be made available immediately.
  • sensor data patches that are physically plausible to one another can be provided for several sensors in this way.
  • the sensor data patches are generated from the sensor data as a function of the at least one property information item with which the sensor data are marked. For this purpose, partial excerpts of those sensor data are selected or used to which the property information is assigned that is predetermined by the at least one property parameter. As a result, only specific sensor data or sensor data filtered according to specific property information can be obtained or queried from the initial domain databases. For example, it can be provided that the sensor data used to generate the sensor data patches must have been recorded in the following context: winter, Mondays, “pedestrians present”. Corresponding property parameters are then specified for this scenario and the sensor data are retrieved and obtained as a function of the specified property parameters. Sensor data patches are then generated from the sensor data obtained by extracting partial sections from the sensor data and storing them as sensor data patches in the database. The sensor data patches generated in this way can then also be marked with the associated property information.
  • a distribution of properties is specified via the at least one property parameter, which the sensor data patches stored in the database are to have.
  • statistical distributions can be specified using mean values and standard deviations. This is done accordingly Generation of the sensor data patches from the received sensor data taking into account the given distribution.
  • the at least one property parameter comprises a regulatory specification.
  • a regulatory stipulation is, for example, a stipulation by a legislator that specifies how a database that is used in quilting to robustize sensor data must be designed or what criteria it must meet. For example, it can be provided that a proportion of sensor data patches is specified that originates from sensor data that depict traffic signs or faces, etc.
  • Another example is a specification for the existence of a specific context that must be mapped by the database created. Such a context can be, for example, the specification that properties or features of a play street, a cycle path and / or a zebra crossing must be mapped in the database using appropriate sensor data patches. In this way, it can be achieved that regulatory requirements, which serve to ensure or increase security, are taken into account when generating the database.
  • a reconstruction quality parameter is obtained, the reconstruction quality parameter specifying an average distance between sensor data and sensor data replaced piece by piece by means of the generated database on the basis of the generated database, and the generation of the sensor data patches being carried out taking into account the reconstruction quality parameter obtained.
  • the average distance can be determined, for example, with the aid of a distance measure that represents a partial section of the sensor data replaced with a sensor data patch and the original partial section in each case linearized as vectors and using a vector standard, for example the L2 standard, determines a distance between these vectors.
  • a distance determined in this way is averaged over a predetermined amount of sensor data so that an average value is formed.
  • the average distance can then be changed, for example, by adding and / or removing sensor data patches to / from the database.
  • a data domain distance parameter is obtained, with a respective proportion of the sensor data patches generated from the plurality of output domain databases depending on the received data
  • Data domain distance parameter is set. This can help one through quilting with the reconstruction data domains formed by the sensor data patches of the database are approximated in terms of their properties to the output data domains mapped in the output domain databases. In simple terms, a distance to an output data domain can be reduced by increasing a proportion of the sensor data from this output data domain when generating the sensor data patches. A domain distance between the respective output data domains and the reconstruction data domain of the sensor data replaced by means of the sensor data patches is determined by means of a suitable domain distance measure. Methods that can be used to determine a data domain distance are described below.
  • the database is optimized with regard to at least one optimization criterion.
  • a nature or a composition of the database is changed iteratively until the at least optimization criterion is met.
  • Parameters that can be changed for optimization are, for example, a size of the sensor data patches, a number of sensor data patches in the database, a respective share of sensor data from different output data domains or output data domain databases, a strength of the influence of property information when generating the sensor data patches and / or a Selection of the distance measure (s) and / or selection method when generating the sensor data patches for the database and / or when selecting sensor data patches stored in the database during quilting.
  • One optimization criterion is, for example, a predetermined domain distance between the output data domains mapped via the output domain databases and the reconstruction data domain generated by means of quilting with the replaced sensor data.
  • a data domain distance measure for example, statistical properties or statistical characteristic values of data sets formed from sensor data of a known data domain and a data set of the reconstruction data domain formed from replaced sensor data can be determined and compared with one another.
  • camera images for example, color value histograms over the camera images and replaced camera images or the respective partial excerpts or sensor data patches can be compared with one another.
  • a data domain and the reconstruction data domain can then be based on the respectively determined statistical properties or characteristic values of the associated Data sets are compared with one another and from this the data domain distance measure can be determined.
  • the data domain distance measure between data sets from two data domains can be calculated, for example, from a difference between the respectively associated statistical characteristic values.
  • the data domain distance measure is determined on the basis of a respective distribution of features in the data sets of the data domains formed from the original or replaced sensor data.
  • features are extracted from the respective sensor data or the partial excerpts from the sensor data, the statistical distributions of which are then compared with one another in pairs for a data set of a data domain and a data set of the reconstruction data domain.
  • the data domain distance measure is then determined from the comparison result, for example by forming the difference between statistical characteristic values of the respective statistical distributions.
  • the features can originate, for example, from dimension reduction methods and / or be provided by kernel functions and / or feature maps of a (deep) neural network. Classification results of a (deep) neural network can also be used as features.
  • the statistical distributions of the features are then compared with one another for the data sets of the data domains by means of statistical characteristic values, such as expected values etc., and defined as the data domain distance between pairs of data sets of the data domains.
  • Cross Domain Retrieval a sampled k-NN confusion
  • three sets of features are created, one set of features each from the original or replaced data sets of the two data domains to be compared, as well as a mixed set of features from both data domains. All three records are drawn randomly from the respective data of the data records (e.g. 1000 samples per record). Then, for each sample of the data domain-specific records, the k-nearest neighbors in the mixed record are searched for, and how many of them originate from the other data domain are counted. The determined numbers can then be used as a data domain distance measure.
  • a reconstruction quality can also be an optimization criterion.
  • a reconstruction quality here denotes a similarity or a distance between sensor data and the associated replaced sensor data.
  • An optimization criterion can also be a robustness effect. Such a robustness effect quantifies a robustification that was achieved by means of quilting with the help of the created database of sensor data of a target data domain that is provided to a neural network or a selection of neural networks (in particular to provide a function for automated driving).
  • An optimization criterion can also be a performance effect.
  • a performance effect can quantify an effect of sensor data robustized by quilting with the aid of the database generated.
  • both adversarially disturbed sensor data and sensor data that has been replaced piece by piece with the help of the database are fed to a (deep) neural network or a selection of (deep) neural networks (in particular to provide a function for automated driving) and the effect is obtained by comparing the results quantified.
  • a functional performance accuracy of the prediction, intersection over union, false-positive or false-negative detection rate, etc.
  • a generalization ability functional quality under previously unseen contexts
  • the respective measured values for a neural network or a selection of neural networks are measured with the supply of sensor data as well as sensor data that is replaced piece by piece, and the performance effect is quantified in this way.
  • Domain invariance in the reconstruction can also be an optimization criterion.
  • a domain invariance is characterized here by the smallest possible distance between different reconstruction data domains among different output data domains.
  • An optimization criterion can also be a domain invariance for a selection of (deep) neural networks, i.e. the same performance and robustness of (deep) neural networks (especially for providing a function for automated driving) on different reconstruction data domains formed from replaced sensor data.
  • the optimization criterion is or is being determined by regulatory requirements. This can be, for example, a minimum size of the sensor data patches, a minimum robustness that is achieved during quilting with the aid of the database generated and / or a maximum latency when using the database when reconstructing the sensor data, that is to say during quilting.
  • the at least one optimization criterion comprises a runtime property of the database during quilting.
  • a runtime property can be improved, for example, in that typical combinations of sensor data patches, that is to say that frequently occur when quilting sensor data, are determined in advance and kept ready for the reconstruction or quilting of the sensor data.
  • Search indices can also be set up, which accelerate a quick search of the database for a sensor data patch with the smallest distance to a partial section of the recorded sensor data.
  • the database can be hashed efficiently for parameters that are highly relevant for the reconstruction.
  • the hashing tables determined are stored in the database so that a subsequent search in the database can be accelerated.
  • Voronoi cells can be determined for the reconstruction, i.e. partial sections in the sensor data which are always replaced by a certain sensor data patch during the reconstruction or during quilting.
  • the specific Voronoi cells or specific subsections are marked accordingly in the database and / or stored separately.
  • additional information is stored in the database that describes or specifies the properties of the database or properties of sensor data patches stored in the database. This makes it easier to use the database for quilting.
  • the properties are stored in the form of metadata.
  • a distance measure is determined which defines a distance between the partial sections to be replaced in the sensor data and the sensor data patches, and is stored in the database. This distance measure defines how a distance between the partial sections and the sensor data is to be measured. It is provided here that the same distance measure is used both when the database is generated and when the database is subsequently used when quilting or reconstructing sensor data.
  • the partial sections of the sensor data and the stored sensor data patches can be displayed as vectors.
  • Vector norms such as the LP metrics, for example the L2 norm, are then particularly suitable as distance measures.
  • a distance cannot be defined on the basis of specific sensor data patches, but rather on the basis of a selection of sufficiently similar sensor data patches.
  • a selection method for selecting sensor data patches from the database is established and stored in the database. This selection method is queried from the database when the database is used and then used to select the sensor data patches.
  • optimization of the database with regard to a predefined optimization criterion can be supported, since the optimization can also include a choice or a change in the selection method.
  • An exemplary selection method is “Nearest Neighbor”, which means that the sensor data patch that is closest in terms of a distance is always selected.
  • a sensor data patch can be selected as a function of property information (see above) and / or using probabilistic methods.
  • a sensor data patch can be selected from a set of sensor data patches that are sufficiently close with respect to a distance, for example by means of the Monte Carlo method. Which sensor data patches are taken into account is defined, for example, via a predetermined limit value for the distance.
  • a data domain distance between respective output data domains of the output domain databases and a reconstruction data domain formed by means of quilting based on sensor data patches stored in the database is determined and stored in the database.
  • databases for specific data domains of sensor data can be generated and made available in a targeted manner.
  • An assessment of a database that has been created can also take place via the data domain distance.
  • an optimization of the database with regard to an optimization criterion that contains the smallest possible data domain distance to one or more output data domains can be supported.
  • the domain distance measures already described above can be used as the domain distance measure.
  • an assistance system for a vehicle comprising at least one robustification device, with a computing device and a storage device, the robustification device being set up to do so by means of a method to obtain and / or receive database generated according to one of the described embodiments and to store it in the storage device, and to replace acquired sensor data piece by piece with the help of the sensor data patches stored in the database by means of quilting, so that the replaced sensor data are robustized against adverse disturbances.
  • the assistance system has a control device, the control device being set up to provide at least one function for automated or partially automated driving of the vehicle and / or for perception of the surroundings, the assistance system being set up to perform the at least one function to supply the replaced sensor data. Based on the supplied, replaced sensor data, the function generates at least one control signal and / or evaluation signal and makes this available.
  • Fig. 1 is a schematic representation of an embodiment of the device for
  • Fig. 2 is a schematic representation to illustrate embodiments of the
  • the device 1 shows a schematic representation of an embodiment of the device 1 for generating and providing a database 40 with sensor data patches 60 stored therein for use in quilting.
  • the device 1 comprises a data processing device 2, which comprises a computing device 3 and a storage device 4.
  • the device 1 carries out the method described in this disclosure for generating a database 40 with sensor data patches 60 stored therein for use in quilting.
  • Parts of the device 1, in particular the data processing device 2 can be designed individually or collectively as a combination of hardware and software, for example as program code that is executed on a microcontroller or microprocessor.
  • the data processing device 2 provides several output domain databases 10, 11.
  • the initial domain databases 10, 11 are in particular stored in the storage device 4.
  • the data processing device 2 can access the multiple output domain databases 10, 11 if these are provided, for example, by an external device.
  • Starting domain databases 10, 11 are each stored with sensor data 20, 21 that were recorded by sensors calibrated to one another.
  • the respective sensor data 20, 21 are, in particular, sensor data 20, 21 of a plurality of sensors which are calibrated with respect to one another in terms of time and location.
  • the sensors can be, for example, a camera and a lidar sensor calibrated for this purpose. However, other and in particular further sensors can also be provided. It can be provided that the
  • the data processing device 10, 11 receives the output data domain databases 10, 11 or the sensor data 20, 21 stored therein for this purpose via an input interface 5 and stores them in the storage device 4.
  • the computing device 3 receives, in particular receives, for example via the input interface 5, a number parameter 30 and a size parameter 31. Furthermore, the computing device 3 retrieves sensor data 20, 21 from the output domain databases 10, 11.
  • the computing device 3 initializes a database 40 in the storage device 40 and uses the received sensor data 20, 21 to generate a number of sensor data patches 60 defined by the number parameter 30, each with a size predetermined by the size parameter 31. Provision is made here for the number to be divided proportionally between the multiple output data domain databases 10, 11.
  • a sensor data patch 60 is in particular a partial section from the received sensor data 20, 21. If the sensor data 20, 21 is, for example, a camera image, then a sensor data patch 60 corresponds to an image section from the camera image, for example with 8x8 picture elements (pixels). If the sensor data 20, 21 are, for example, two-dimensional lidar data, then a sensor data patch 60 corresponds to a partial section from the lidar data, for example with 8 ⁇ 8 data points.
  • the generated sensor data patches 60 are stored in the database 40 by the computing device 3.
  • Sensor data patches 60 which are composed of sensor data 20, 21 that correspond to one another in terms of time and location (for example a camera image and herewith corresponding lidar data) are generated, in particular, are stored linked to one another in the database 40.
  • the database 40 generated is provided by the computing device 3; for example, the database 40 can be output as a data packet via an output interface 6.
  • the database 40 is then fed in particular to a robustification device 70 of an assistance system 300 of a vehicle (not shown), which replaces sensor data 72 acquired by means of quilting piece by piece, so that the replaced sensor data 73 are robustized against adverse disturbances.
  • the database 40 is loaded into a storage device (not shown) of the robustification device 70, for example.
  • the replaced sensor data 73 can be fed to a control unit 51 of the assistance system 300.
  • the control device 51 provides a function for automated and / or partially automated driving of the vehicle and / or for perception of the surroundings.
  • the replaced sensor data 73 are fed to the function. Based on the replaced sensor data 73, the function generates, for example, a control signal 52 or an evaluation signal 53.
  • the sensor data 20, 21 in the multiple output domain databases 10, 11 are at least partially marked or marked with at least one property information item 12, 13, at least one property parameter 32 being received from the computing device 3, for example via the input interface 5 becomes.
  • the generation of the sensor data patches 60 for the database 40 is then carried out taking into account the obtained at least one property parameter 32 and the at least one property information item 12, 13. In particular, this can be used to determine which property information 12, 13 is accepted as patch property information 14 or is assigned to the sensor data patches 60 generated in the database 40.
  • the at least one property parameter 32 can be used to specify which properties the generated sensor data patches 60 must have in order, for example, to map predefined property distributions (with regard to a context, weather, critical situations, etc.) in the database 40.
  • the at least one property parameter 32 comprises a regulatory specification.
  • the computing device 2 has a
  • the reconstruction quality parameter 33 specifies an average distance between sensor data 72 and sensor data 73 replaced piece by piece by means of quilting on the basis of the database 40 generated, and the sensor data patches 60 are generated taking into account the reconstruction quality parameter 33 obtained.
  • the sensor data 72, 73 for determining the average distance or a reconstruction quality are fed to the computing device 3.
  • the average distance or the reconstruction quality can also be determined by the robustification device 70 and fed to the computing device 3.
  • an average distance or a reconstruction quality is determined for each iteration step of an optimization method for optimizing the database 40.
  • the reconstruction quality is then compared with the reconstruction parameter 33 obtained and the procedure for generating the sensor data patches 60 is changed as a function of a deviation.
  • the size and / or the number of the sensor data patches 60 and / or a proportion of the output data domains etc. can be changed for optimization.
  • the computing device 2 receives a data domain distance parameter 34, for example via the input interface 5, a respective proportion of the sensor data patches 60 generated from the multiple output domain databases 10, 11 being determined as a function of the data domain distance parameter 34 obtained. If a reconstruction data domain formed from the sensor data patches 60 after quilting is to have a smaller / larger data domain distance to one of the output data domains mapped in the output domain databases 10, 11, the associated proportion of the output data domain is correspondingly reduced / increased when the sensor data patches 60 are generated.
  • the database 40 is optimized with regard to at least one optimization criterion 35.
  • the optimization criterion 35 can include, for example, one or more of the following criteria: a predetermined domain distance, a reconstruction quality, a robustness effect, a performance effect, a domain invariance.
  • a predetermined domain distance for example, the size and / or the number of the sensor data patches 60 and / or a proportion of the output data domains 10, 11 and / or an influence of property information 12, 13 or a property parameter 32 and / or distance dimensions and / or selection method etc. change.
  • the at least one optimization criterion 35 includes a runtime property of the database 40 during quilting.
  • the computing device 2 can, for example, efficiently hashing the database 40 compiled from the sensor data patches 60, so that a search can be accelerated during a subsequent quilting in the robustification device 70.
  • FIG. 2 shows a schematic representation to clarify embodiments of the method for generating and providing a database 40 with sensor data patches stored therein for use in quilting.
  • a method step 100 the method for generating a database 40 with sensor data patches stored therein for use in quilting is carried out in one of the embodiments described in this disclosure.
  • the initial domain databases 10, 11, 15, 16 are used here. These include, for example, sensor data with basic truth, sensor data without basic truth and simulated sensor data for different data domains.
  • one property parameter with at least one regulatory stipulation 17 is taken into account.
  • a proportion of sensor data patches is specified that originates from sensor data that depict traffic signs or faces, etc.
  • a regulatory stipulation 17 for the existence of a specific context e.g. play street, zebra crossing, etc.
  • the result of method step 100 is the database 40 with the sensor data patches contained therein.
  • the database 40 is loaded into a storage device of a robustification device 70 of an assistance system, which is arranged in a vehicle 50.
  • the robustification device 70 robustizes sensor data 72 acquired by sensors 71 with the aid of the sensor data patches stored in the database 40 by replacing them piece by piece by means of quilting.
  • the replaced sensor data 73 which are also called reconstructed sensor data are then fed to a (deep) neural network 74, which in particular provides a function for the automated or partially automated driving of the vehicle 50 and / or for driver assistance of the vehicle 50 and / or for a perception and / or perception of the surroundings.
  • the neural network 74 is provided by means of a control device 51 (cf. FIG. 1) of the assistance system of the vehicle 50.
  • the neural network 74 generates at least one control signal 52 (FIG. 1) and / or evaluation signal 53 (FIG. 1) and provides this.
  • an actuator system of the vehicle 50 can be controlled and / or based on the at least one evaluation signal, a further (higher-value) evaluation can take place.
  • the (deep) neural network 74 was previously trained in a training step 200 on the basis of training data obtained from the initial data domain databases 10, 11, 15.
  • the advantage of the method and the device is that a database can be generated and provided as a basis for a safety and robustness-oriented reconstruction of sensor data by means of quilting.

Abstract

L'invention concerne un procédé de génération d'une base de données (40) dans laquelle sont stockées des pièces de données de capteur (60) destinées à être utilisées dans le matelassage, dans lequel procédé : une pluralité de bases de données de domaine initial (11, 12, 15, 16) sont fournies et/ou accédées ; des données de capteur (20, 21), qui ont été détectées au moyen de capteurs qui sont étalonnés les uns avec les autres, sont respectivement stockées dans la pluralité de bases de données de domaine initial (11, 12, 15, 16) ; des données de capteur (20, 21) sont obtenues à partir de la pluralité de bases de données de domaine initial (11, 12, 15, 16) ; un paramètre de nombre (30) et un paramètre de taille (31) sont obtenus ; un certain nombre de pièces de données de capteur (60) déterminées à partir des données de capteur obtenues (20, 21) par le paramètre de nombre (30) sont générées dans chaque cas avec une taille prédéterminée par le paramètre de taille (31) et stockées dans la base de données (40) ; le nombre est distribué proportionnellement à la pluralité de bases de données initiales de domaine de données (11, 12, 15, 16), et la base de données générée (40) est fournie. L'invention concerne en outre un dispositif (1) pour générer une base de données (40) dans laquelle sont stockées des pièces de données de capteur (60), destinées à être utilisées dans le matelassage.
EP20829549.3A 2019-12-17 2020-12-10 Procédé et dispositif pour générer et fournir une base de données dans laquelle sont stockées des pièces de données de capteur destinées à être utilisées dans le matelassage Pending EP4078433A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102019219924.0A DE102019219924B4 (de) 2019-12-17 2019-12-17 Verfahren und Vorrichtung zum Erzeugen und Bereitstellen einer Datenbank mit darin hinterlegten Sensordatenpatches zur Verwendung beim Quilting
PCT/EP2020/085652 WO2021122340A1 (fr) 2019-12-17 2020-12-10 Procédé et dispositif pour générer et fournir une base de données dans laquelle sont stockées des pièces de données de capteur destinées à être utilisées dans le matelassage

Publications (1)

Publication Number Publication Date
EP4078433A1 true EP4078433A1 (fr) 2022-10-26

Family

ID=74068256

Family Applications (1)

Application Number Title Priority Date Filing Date
EP20829549.3A Pending EP4078433A1 (fr) 2019-12-17 2020-12-10 Procédé et dispositif pour générer et fournir une base de données dans laquelle sont stockées des pièces de données de capteur destinées à être utilisées dans le matelassage

Country Status (3)

Country Link
EP (1) EP4078433A1 (fr)
DE (1) DE102019219924B4 (fr)
WO (1) WO2021122340A1 (fr)

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11214268B2 (en) 2018-12-28 2022-01-04 Intel Corporation Methods and apparatus for unsupervised multimodal anomaly detection for autonomous vehicles
US11507084B2 (en) * 2019-03-27 2022-11-22 Intel Corporation Collaborative 3-D environment map for computer-assisted or autonomous driving vehicles

Also Published As

Publication number Publication date
DE102019219924A1 (de) 2021-06-17
DE102019219924B4 (de) 2022-02-17
WO2021122340A1 (fr) 2021-06-24

Similar Documents

Publication Publication Date Title
EP3983936A1 (fr) Procédé et générateur servant à créer des données d'entrée perturbées destinées à un réseau neuronal
EP4078238A1 (fr) Procédé et dispositif pour rendre des données de capteur plus robustes à l'égard de perturbations indésirables
DE102008036219A1 (de) Verfahren zur Erkennung von Objekten im Umfeld eines Fahrzeugs
DE102019208735B4 (de) Verfahren zum Betreiben eines Fahrassistenzsystems eines Fahrzeugs und Fahrerassistenzsystem für ein Fahrzeug
DE102019219924B4 (de) Verfahren und Vorrichtung zum Erzeugen und Bereitstellen einer Datenbank mit darin hinterlegten Sensordatenpatches zur Verwendung beim Quilting
WO2021122337A1 (fr) Procédé et appareil de reconnaissance de suppression d'un domaine de données de capteur à partir d'un domaine de données de référence
EP4049186A1 (fr) Procédé pour robustifier un réseau neuronal vis-à-vis de perturbations antagonistes
WO2021122339A1 (fr) Procédé et dispositif permettant de fabriquer un réseau neuronal plus robuste par rapport à des perturbations antagonistes
EP3973466A1 (fr) Procédé pour rendre un réseau neuronal plus robuste de manière spécifique à son fonctionnement
EP3973458A1 (fr) Procédé pour faire fonctionner un réseau neuronal profond
DE102019213459A1 (de) Verfahren zum Komprimieren eines Neuronalen Netzes
DE102019217952A1 (de) Verfahren und Vorrichtung zum Bereitstellen eines Trainingsdatensatzes zum Trainieren einer KI-Funktion auf eine unbekannte Datendomäne
DE102019217951A1 (de) Verfahren und Vorrichtung zum Bestimmen einer Domänendistanz zwischen mindestens zwei Datendomänen
DE102019219926A1 (de) Verfahren und Vorrichtung zum Trainieren eines Neuronalen Netzes
DE102020213058A1 (de) Verfahren und Vorrichtung zum teilautomatisierten oder vollautomatisierten Steuern eines Fahrzeugs
DE102022201161A1 (de) Objektklassifizierung mit einem einstufigen metabasierten Objektdetektor unter Verwendung von Klassenprototypen
DE102020203705A1 (de) Plausibilisierung der Ausgabe neuronaler Klassifikatornetzwerke anhand von Zusatzinformationen über Merkmale
DE102021206377A1 (de) Robusteres überwachtes Training für Bildklassifikatoren
DE102022104399A1 (de) Verfahren und Prozessorschaltung zum Ermitteln von Trainingsdatensätzen für ein Training eines Modells des maschinellen Lernens
EP3985565A1 (fr) Procédé et dispositif de vérification d'un système de traitement d'informations basé sur l'ia utilisé lors de la commande partiellement automatisée ou entièrement automatisée d'un véhicule
DE102021214552A1 (de) Verfahren zum Evaluieren eines trainierten tiefen neuronalen Netz
DE102020208737A1 (de) Verfahren und Vorrichtung zum Bewerten und Zertifizieren einer Robustheit eines KI-basierten Informationsverarbeitungssystems
DE102022208480A1 (de) Verfahren zum Evaluieren eines trainierten tiefen neuronalen Netzes
DE102022110233A1 (de) Computer-implementiertes verfahren zur auswahl von trainingsdatensätzen aus einer vielzahl von datensätzen
DE102019213458A1 (de) Verfahren zum Komprimieren eines Neuronalen Netzes

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20220718

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS