DE102020210188A1 - VEHICLE ENVIRONMENT RECOGNITION DEVICE - Google Patents

Abstract

A vehicle environment detection device for a host vehicle 1, which contains an optical sensor 2, which is communicatively connected to the vehicle environment detection device 10, wherein the vehicle environment detection device 10 is configured to output a sequence of image data that at least shows the future distance between the host vehicle 1 and the one or the contains several detected objects O1, O2.

Description

The present disclosure relates to a vehicle environment recognition device for a host vehicle, which in particular reduces the complexity of e.g. B. the hardware used is reduced, which provides increased accuracy and reliability of the output and which is generally applicable because it can be easily integrated into vehicles or mono cameras.

background

Autonomous vehicles rely on sensors to detect the environment (environment) containing objects. Typical software architectures of autonomous vehicles merge information from multiple sensors such as e.g. B. Cameras, LiDAR devices and radar devices to perceive the environment accurately. Cameras are used to detect object types such as B. vehicles, traffic signs, pedestrians, etc. to detect and classify. LiDAR devices are used to measure the position and orientation of objects, while radar devices are used to measure speed.

However, it is a disadvantage that the vehicle requires input from several different sensors, as described above, which increases the complexity of the hardware architecture as well as the software-based data processing.

Further describes US 2019/0 317 519 A1 a system architecture that uses a three-dimensional (3D) ranging module that calculates a range to an object based on a 3D representation that has a dimension that best matches the dimensions (e.g., a length, a width, or an area (e.g., length by width) of a two-dimensional (2D) bounding box. The distance estimation module uses a search algorithm such as B. a greedy search or a binary search as a basis. The distance to the object at the current time can only be estimated after estimating the object's orientation and 3D size.

WO 2019/177562A1 describes a system architecture that uses machine learning models to detect an object in an image and to determine the distance to the detected object at the current time.

However, enabling a further increased reliability and accuracy of the object detection and the information provided by a vehicle environment recognition device is still needed, especially when less complex hardware systems e.g. B. Systems that rely on the input (only) of a mono camera instead of multiple vehicle sensors can be used.

problem and solution

An object of the disclosure described here is to provide a solution that improves in particular the accuracy and the reliability of the output of a vehicle environment detection device that can be used in vehicles, preferably autonomous vehicles, and that allows the use of a less expensive and less complex hardware architecture that can be easily integrated.

In particular, the present disclosure proposes a technique that can be used to determine the posture (i.e. position, orientation and their derivatives) and extent of surrounding objects from a single optical sensor, e.g. B. a (mono) camera, to estimate and predict at the same time. One benefit is reduced complexity and cost compared to existing approaches that require multiple sensors (e.g., camera, LiDAR device, radar) to potentially achieve the accuracy and reliability of the present disclosure.

In particular, the following aspects are created:

A vehicle environment detection device for a host vehicle, which may include an optical sensor communicatively connected to the vehicle environment detection device. The optical sensor can be an imaging semiconductor, i. H. an image sensor such as a CCD or CMOS sensor, or it may be a camera further including optical lenses and the like. Most preferably, the optical sensor is a single sensor or mono camera. Using a mono camera can save costs and reduce system complexity, e.g. B. in comparison to a stereo camera or a combination of different sensors such. B Lidar or radar.

The vehicle environment detection device may include or be integrated in at least one computing unit and may be configured to receive image data at a current time t from the optical sensor mounted on the host vehicle. The expression "current time t" can preferably indicate the actual time, while a time t - n belongs to the past and the times t + k and t + Dt are future times. The image data can preferably be received at the current time t the. Preferably, the image data processing such. B. start the object detection at the current time t.

The vehicle surroundings recognizer may be further configured to detect objects in the received image data, where an object may refer to people, pedestrians, animals, vehicles, roadways, lane markings, hazards, and objects in general with respect to the position of the vehicle. It may further be configured to determine at least the distance between a (or each) detected object and the host vehicle and the speed of a (or each) detected object. The distance and speed are determined for the current time t, which preferably means at the time the image was captured/received/its processing started.

Furthermore, the vehicle environment detection device can be configured to predict at least one future distance of a (or each) detected object from the host vehicle. Preferably, several future distance values are determined for a (or each) detected object for a predefined time interval starting from the current time t to a future time t+k. More preferably, a set of a future set/sequence of distance values for a detected object is generated, wherein the future time interval is segmented into a predefined number of points in time, for each of which a distance value is generated. In other words, multiple distance values are preferably determined for each detected object at different times between time t and time t+k.

Further, the vehicle environment recognizer may be configured to compensate for the movement of the host vehicle based on a processing and communication time lag Dt at time t+Dt. The time interval Dt includes or is the time required for the processing of the object detection and the prediction as well as for the signal transmission between sub-units of the vehicle environment recognition device and/or in a network (latency). The time interval Dt can be measured by the vehicle environment detection device, another vehicle controller or a dedicated sub-unit in real time, it can be estimated based on previous comparable scenarios or it can be calculated/determined when the relevant parameters are known or estimable. The compensation makes it possible, in particular, to recalculate the change in position of the host vehicle during the short time interval from time t to time t+Dt.

Furthermore, the vehicle surroundings recognition device may be configured to output a sequence of images containing image frames in the interval from time t (preferably the interval starts at t+Dt) to t+k. The images may include the position of the host vehicle and the (Cartesian) positions as well as the (Euclidean) distances, dimensions, etc. of the one or more detected objects. The distances etc. may take into account the compensated/corrected position of the host vehicle at time t+Dt. The output can be delivered to a route planning device of an autonomously driving host vehicle, to a navigation system and/or to the driver of the host vehicle.

The present aspect makes it technically advantageous that high accuracy and reliability of the output information about detected objects can be achieved even using only a mono camera (a single optical sensor). The information about the positions and further information of the detected objects in the (near) future enable a more precise route planning, which e.g. B. increases security in terms of avoiding collisions. Accuracy is also increased due to the position correction of the host vehicle position changing during the time interval of receiving the one or more images from the optical sensor to complete the processing of the one or more images. The present device can account for this shift in position for further improved accuracy and reliability. The complexity of the system is significant compared to host vehicle systems that require multiple different input sources for detecting and recognizing the environment of the host vehicle, e.g. B. reduced by the complexity of calibration between multiple sensors. Furthermore, z. B. the integration into existing vehicle cameras, in particular mono cameras, due to the technical advantage that a calibration of camera parameters or the like is not necessary, can be easily carried out.

In addition, the vehicle environment detection device can include a detection unit, which can be connected in terms of communication technology at least to the optical sensor of the host vehicle and a prediction unit. The prediction unit, which is preferably part of the vehicle environment detection device, can be communicatively connected at least to the detection unit and a host vehicle compensation unit, which can be part of the vehicle environment detection device. The host vehicle compensation unit may be provided with at least one processing and communication time delay determination unit which is part of another Device may be, a host vehicle driving parameter unit, which may be part of another device and the data from sensors of the host vehicle such. speed sensors, yaw sensors, position sensors, engine parameter sensors, etc., can be communicatively connected to the prediction unit and an output image data unit, which can be part of the vehicle environment recognition device.

Furthermore, the detection unit can be configured to receive the image data from the optical sensor at a current time t, to detect one or more objects in the image data and to output object information data about the one or more detected objects to the prediction unit, the object information data each of the one or of the multiple detected objects contain at least the distance to the host vehicle and the speed at the current time t. Furthermore, the object information data may further include the object's orientation, its bounding box coordinates, and its dimensions, among other things.

The prediction unit may be configured to predict future object information data about the one or more detected objects based at least on the object information data received from the detection unit. The future object information data of each of the one or more detected objects may include at least future distance values to the host vehicle between the current time t and a future time t+k. Furthermore, the future object information data may further include the object's orientation, its bounding box coordinates, and its dimensions, among other things.

The host vehicle compensation unit may be configured to correct the position of the host vehicle at a time t+Dt based on host vehicle running data received from the host vehicle running parameter unit and the processing time lag Dt received from the processing time lag determination unit. The host vehicle driving data may include the host vehicle's speed, yaw, and the like. The processing time delay determination unit may be configured to measure, estimate, calculate, and/or determine the time interval Dt (the latency).

The output image data unit may be configured to receive the corrected position of the host vehicle at time t+Dt, the object information data output by the detection unit and the future object information data output by the prediction unit, and a sequence of image data from time t + Dt at time t + k in a time interval from time t to t + Dt. In other words, the time interval from time t to t+Dt is the time interval during which the processing/generation of the future image data/the object information data is performed. The image data can contain at least all detected objects, which contain their (future) object information data as well as the position of the host vehicle.

The technical advantages explained above are also achieved for the preferred aspect explained above. In addition, the structure of the communication-related units allows the received image data to be flexibly processed in various application scenarios. For example, compared to the prior art, the object distance estimation can be separated from the object detection and thus two specialized and therefore more accurate units configured/trained for a specific purpose can be employed. Furthermore, the units may be distributed across different locations in a system that may include remote computing units, the computing unit in the host vehicle, and the like, allowing the hardware architecture to be adapted to different scenarios.

Furthermore, the image data received by the detection unit at the current time t may comprise/may be formatted as a sequence of image frames containing multiple frames from a time t - n to the current time t, where t - n is a Time in the past relative to the current time t. The number of frames can be defined by a time (or a time difference) or by a predefined value, e.g. eg it can be defined such that a time frame is set for every millisecond or shorter or longer, or it can be defined such that in the time interval t - n to t uses a predefined number (regardless of the duration of the interval) of frames will/should exist.

Using sequences of images/image frames as an input to the device provided at time t improves the database for detection of objects and the prediction processing for computing the images of the future time interval. As a result, increased accuracy and increased reliability can be achieved.

The object detection, which can preferably be part of/integrated into the detection unit, can be a trained machine learning (ML) model and/or a trained artificial model Use human intelligence (AI model) to detect an object. The one or more models may be contained in the detection unit or the detection unit may be communicatively connected to the models if those are stored separately from the detection unit (e.g. in a web-based server/processing unit). The machine learning model and/or the artificial intelligence model of the detection unit can be trained as the object information data for each detected object, the type of the detected object, the orientation of the detected object with respect to a host vehicle-based coordinate system, a bounding box of the detected object and the Coordinates in the host vehicle-based coordinate system to determine the dimensions (or size) of the detected object, the distance to the host vehicle and/or the speed of the detected object.

As explained above, the type of an object can be e.g. B. categorize the objects into pedestrian, car, truck, obstacle or the like. The orientation of a detected object can be detected in relation to the host vehicle and can e.g. B. the direction of its axis, z. B. specify the longitudinal axis. The term bounding box may preferably be a rectangular or a square area surrounding the outside dimensions of the detected object in the image plane (as close as possible without intersecting the object's outside boundary lines). The bounding box can be defined by the coordinates of its upper left corner and lower right corner. The dimensions can include the length, width, height, etc. of the detected object, and the size can indicate the area or one of the dimensions.

The distance may preferably be defined based on a coordinate system, which may originate at the host vehicle's optical sensor or at any other well-defined position of the host vehicle.

The combination/set of parameters described above enables the position and behavior of a detected object to be detected and determined in a very accurate manner, while at the same time the number of parameters is relatively low, which reduces the computational effort and data traffic. In other words, the entire device can achieve high accuracy/precision with a low complexity of the required hardware/software architecture.

Furthermore, the prediction preferably included/integrated in the prediction unit may be a trained machine learning model and/or a trained artificial intelligence model for performing e.g. B. use a time series analysis to predict the future information data. The one or more models may be included in the prediction engine, or the prediction engine may be communicatively coupled to the models when stored separately from the prediction engine (e.g., in a web-based server).

The machine learning model and/or the artificial intelligence model can be trained as the future object information data for each detected object and for a sequence of image frames in a time interval between the current time t or t+Dt and a future time t+k determine the orientation of the detected object with respect to a host vehicle-based coordinate system, a bounding box of the detected object and the coordinates in the host vehicle-based coordinate system, the dimensions (or size) of the detected object, the distance to the host vehicle and/or the speed of the detected object.

Providing the images for the predicted future behavior, position, and/or the like of each detected object enables superior accuracy and safety of route planning or host vehicle driving.

Furthermore, the machine learning model and/or the artificial intelligence model may contain a convolutional neural network for detection and a recurrent neural network for prediction.

The use of specialized neural networks, specifically designed for the intended tasks, allows for even more improved precision and reliability of the output data/results.

Furthermore, the detection of the distance of each detected object to the host vehicle at a current time t can be performed by a separate step, which can preferably mean that the detection unit and the prediction unit can be separate units/use different ML/AI models. The determination of the distance may be performed by a machine learning model and/or an artificial intelligence model used to determine the distance from the input image data received from the host vehicle's optical sensor at the current time t are received, are trained, wherein the received input image data can preferably contain a sequence of image frames with frames from a past time t - n to the current time t.

Furthermore, the data related to the object detection and the prediction and the compensated host vehicle driving data can be merged in each image frame of a sequence of image frames in a time interval from time t+Dt to future time t+k.

Each image in the time interval from time t+Dt to future time t+k can be transformed into an occupancy grid (OCG or ocg) of the environment of the host vehicle. The transformation can be performed by a dedicated ocg generator unit that can perform mathematical transformations to generate the ocgs based on the (future) object information data. The environment of the host vehicle may preferably include the vicinity of the host vehicle and very preferably at least the area overlapping with the area recorded by the one or more optical sensors.

Furthermore, a sequence of occupancy grid image frames can be output in a time interval from time t+Dt to future time t+k.

The provision of future occupancy grids (a sequence of multiple occupancy grid images arranged in chronological order) allows the driver of the host vehicle to intuitively understand future traffic developments in such a way that she/he can adapt driving, and it enables when the host vehicle becomes autonomous drives to adapt the automatically determined course precisely and reliably, e.g. when the ocgs are input to a route planning device/unit or a navigation device/unit of the host vehicle. Furthermore, evaluating the output can also allow control actions such as e.g. B. automated braking, automated acceleration, automated steering and the like automatically derive.

Further, the host vehicle position compensation unit may receive information on the processing delay time Dt caused by the detection and the prediction processing and the communication delay, and on the actual (linear) velocity and the yaw rate of the host vehicle at the time t, and may calculate/determine corrected position of the host vehicle at a time t+Dt. The calculation can use previously trained scenarios such as e.g. B. using an average value or the like as a basis or it can be measured parameters such as e.g. B. use the computing power of the computing unit, the transmission speed of the communication links and the like to calculate the correct value Dt in each case. Furthermore, the value Dt z. B. by recording the time when the image data was received from the optical sensor and by recording the time when the processed data is available (at a previously defined point/step in the flow of processing). In the previous example, the time difference would be the value Dt.

The speed and yaw rate may be provided by host vehicle sensors that may report to the host vehicle driving information unit, which may provide the data to the claimed device/balance determination step.

The balance determination further increases the precision of the data output because the position error of the host vehicle caused by the time required to process the input data and the data transmission between different units is removed. The compensation may use an estimation of a vehicle's movement using a physics-based vehicle model as a basis. Knowing the initial Cartesian position of the vehicle, e.g. B. (X, Y), as well as the actual speed and the actual yaw rate of the host vehicle, the position (X, Y) can be determined / estimated in the next time step.

Further, the vehicle surroundings recognizer may include an identifier tracker unit communicatively connected to a storage unit capable of storing data of previously detected objects. The identifier tracker unit may be configured to determine, for each of the detected objects, whether the object is a newly detected object or a previously detected object, and the object prediction may be configured to receive information from the identifier tracker unit as to whether the detected object is a newly detected object object or a tracked previously detected object.

If the identifier tracker unit is used, the processing time can be significantly reduced because previously determined parameters/data can be read from memory instead of being re-determined by the trained model, or the training model has at least one defined starting point for determination, which reduces the computational effort reduced.

Furthermore, the vehicle environment detection device (in part) a computer pro A program product comprising computer-readable instructions executable in a computer system including one or more computers, the units being hardware and/or software units. The computers may be located in the host vehicle (installed in the host vehicle) and/or remote from the host vehicle, e.g. B. be provided at networked remote locations. If the vehicle environment recognition device is to be a computer program, the entities discussed above may be software modules/entities stored in a memory space accessible to the one or more computers. The vehicle environment recognizer may also be a combination of hardware and software, and some units may be either software or hardware units.

Furthermore, the units may be communicatively linked via hardware and/or software signaling communication channels and communicatively linked via one or more networks, with the units being arranged either in a single piece of hardware or via different hardware devices containing computing units/hardware in the host vehicle , be distributed.

The entities discussed above may be part of a single entity, or each entity may be a separate entity. The vehicle surroundings detection device can itself be part of a mono camera that contains the optical sensor and is installed in a host vehicle, the camera/vehicle surroundings detection device being connected to the control unit of the host vehicle, e.g. B. its central control computer or the like is connected by communication technology.

A preferred architecture can handle the high-effort computational tasks such as B. Distribute the detection of objects, the determination of parameters of a detected object and the prediction processing to network-based (remote) computing resources, while the host vehicle-based units can perform computing tasks with less effort. This enables fast and reliable processing of the input data. If, however, the reliability is to be increased with regard to avoiding interruptions in mobile data transmission, the units explained above can, as far as technically possible, carry out their processing completely/mainly in the computing unit in the carrier vehicle. The network-based resources can then support data such. B. provide map data from a map server, position information, previously stored driving data or the like. Therefore, the architecture can be flexibly adapted to the intended scenario, which creates another technical advantage.

Furthermore, a computer program is disclosed, in one or more computing units such. B. the one or more computing units of the host vehicle and / or remotely located computing units that are connected to the host vehicle via a network.

Furthermore, a system is disclosed that an optical sensor such. B. a (mono) camera and the vehicle environment detection device can contain at least one of the aspects described above. The system may also include remote computing devices, the host vehicle having a controller, storage devices, and one or more networks communicatively connecting remote computing devices and/or storage devices and the host vehicle.

The device disclosed here and the system disclosed here and the computer program product disclosed here make it possible to use a less complex architecture, in particular with regard to the sensor hardware of the host vehicle, which preferably only relies on a single sensor source for the input of data about the environment of the vehicle ( except general information such as map data or the like provided by computing units located in the network) while the output information is extremely reliable and accurate and even information in the (near) future about detected objects in the environment of the vehicle can provide. Furthermore, a technical advantage is that the disclosed apparatus, system and computer program product do not require knowledge of the camera's intrinsic parameters (such as focal length, etc.), making the attempted solution applicable to any (existing) monocular camera makes, regardless of the camera's gap specification. The integration into existing vehicles or cameras can be carried out easily.

character list

• 1 FIG. 12 schematically shows an example of a system of the present disclosure.
• 2a 12 schematically shows an example configuration of a vehicle surroundings recognition device according to the present disclosure.
• 2 B 12 schematically shows an example configuration of a controller of a host vehicle according to the present disclosure.
• 3 12 shows an example of a model training processing flow, in accordance with the present disclosure.
• 4 12 shows an example of a vehicle environment recognition processing flow according to the present disclosure.
• 5 Figure 12 shows an example of the effect of the leveling processing of the present disclosure.
• 6 12 shows an example of a vehicle environment recognition processing flow according to the present disclosure.
• 7 FIG. 12 schematically shows the processing of an image captured by an optical sensor according to the present disclosure.
• 8a and 8b 10 schematically show the generation of an occupancy grid (OCG) or occupancy grid map according to the present disclosure.

Detailed description of exemplary embodiments

Preferred aspects and examples are described in more detail below with reference to the accompanying figures. The same or similar features in different drawings and examples are denoted by similar reference numbers. It should be understood that the detailed description below refers to various preferred aspects and preferred examples are not intended to limit the scope of the present disclosure.

1 shows a diagram of a system 100, which has a carrier vehicle 1 with an optical sensor 2 (which is shown here as a preferably used mono camera), which is communicatively connected to a computing unit 3 (which is indicated by the bold dashed arrows in the figures), contains. The processing unit 3 is installed in the carrier vehicle 1 . The arithmetic unit 3, which is exemplified by 1 is shown includes a vehicle environment detection device 10 according to the present disclosure and a control unit/device 4 and a storage unit 5. If the vehicle environment detection device 10 is to be embodied as a software program product, it may also be stored in the storage unit 5, which is shown in 1 is not shown. Alternatively, a computing unit can be an integrated part of the control device 4 and/or the vehicle environment detection device 10 . Furthermore, the two devices/units 4, 10 can also be integrated into a single controller or can be separated into more than one or two units. The communication connections/lines/channels, in particular between the sub-units in a device/unit, are not shown in the figures for reasons of legibility. In general, the sub-units are connected in a higher-level unit or device in order to exchange information or signals with one another by means of software or hardware communication interfaces and/or communication lines.

The processing unit 3 is connected to one or more networks 6, which are represented by the schematic cloud in 1 are specified, connected by communication technology; furthermore, one or more remote storage units 7 and one or more remote computing units 8a, 8b are connected to the network 6. The processing units 8a, 8b, which are connected to the network 6, z. B. contain a map server, the navigation data such. B. provides map data for the carrier vehicle 1, and also remote computing units 8a, 8b contain parts of the vehicle environment detection device 10 disclosed here, which is particularly important in the case of computationally intensive operations such. B. the processing performed by AI/ML models is preferred. The number of remote computing units 8a, 8b may be lower or higher than the two specified in 1 are shown. The one or more storage spaces 7 can store any type of driving data or any other data that can support the operation of the vehicle environment detection device 10 described here or the operation of the computing unit 3 in the host vehicle 1 or the like.

2 shows a diagram of the vehicle surroundings detection device 10 and the control device 4. 2a indicates that the vehicle environment recognition device 10 includes a plurality of sub-units (although a single sub-unit or more or fewer sub-units than the number of sub-units shown can also be used) which, as will be described further below, include a detection unit 11, a prediction unit 12, a host vehicle compensation unit 13 and an output unit 14 and an OCG generation unit 15 are included. Preferably, the vehicle environment recognizer 10 may further include a processing time delay determination unit 16 and an identifier tracker unit 17 which can receive data from the in-vehicle storage unit 5 or the remote storage unit 7 .

The vehicle environment recognizer 10 may include its own computing unit (not illustrated) and may have its own storage unit (not illustrated) when the vehicle environment recognizer 10 is a separate hardware unit. The same applies to the control device 4. Furthermore, the control device 4 can also be merged with the vehicle environment detection device 10, as described above.

The controller 4 includes one or more sub-units such. B. a host vehicle control unit 41, a sensor data processing unit 42, a brake control unit 43, an engine control unit 44 and the like. In addition, one or more movement plan generation and movement plan provision units 45 can be provided in the previously described control device 4 or in a computing unit 8a, 8b, which is arranged remotely from the carrier vehicle 1.

In addition, the control device 4 can contain a host vehicle driving parameter unit 46 that receives data from sensors (not shown) of the host vehicle 1 or other sources that are not shown, such as e.g. B. GPS signal data or the like.

The host vehicle control unit 41 may e.g. B. be configured to perform an autonomous driving of the host vehicle 1, and can receive an input or control instructions to other sub-units such. B. the sensor data processing unit 42, the host vehicle driving parameter unit 46 can receive, and can provide driving-related data to the host vehicle control unit 41. Further, the host vehicle control unit 41 can control the brake control unit 43 and the engine control unit 44, which can control the brakes and the one or more engines of the host vehicle 1, respectively. A movement plan generation/output unit 45 can be integrated into the host vehicle 1 or can be integrated into the one or more remote computing units 8a, 8b and can perform autonomous driving by setting the route that the autonomously driven/controlled host vehicle 1 should follow, support. The route can be provided to the host vehicle control unit 41 . The carrier vehicle 1 or its computing unit 3 or its control device 4 can also contain information such as B. map data from a map server remotely received over the network 6, such as. B. shown by the reference numeral 8a or 8b. It can also receive any type of previously stored data from the remote storage unit 7 .

The detection unit 11 as well as the prediction unit 12 can use one or more AI and/or ML models (which are not shown) according to the schematic flowchart 3 be trained included.

3 shows the basic course of the training/learning process of the one or more models in such a way that they can be used for a predefined purpose. As an example of this training shows 3 a flow chart of the training steps and the illustrated flow chart is shown in combination with the training unit 50, which performs the following training steps: B. lidar sensors, radar sensors, optical sensors, GPS data, etc. are entered into a process / unit 51 for data labeling and data preparation. The data are marked and prepared (processed, compared, etc.) in such a way that they can be used for training, e.g. B. a neural network can be used. The labeling and preparation of the data can be performed in accordance with principles/process steps already known in the art. Then the prepared data is entered into the learning process/unit 52 as shown in FIG 3 is shown. In the learning unit 52 the data is used to learn the model and thereafter the model is checked as is already known in the art. Model learning and model validation are repeated as long as necessary to arrive at the required/desired precision of the trained model. This can be predetermined, e.g. B. by specifying the number of iterations/repetitions. After the learning process, the model is sufficiently/well trained to be used as a detector model or a predictor model and is output from the training unit 50 .

4 FIG. 1 also shows a basic flow of the processing carried out in the previously described vehicle environment detection device 10 according to FIG 2a and/or the system 100 contained in 1 shown is executed. Special shows 4 that an optical sensor 2, which is preferably a mono camera in order to reduce the system complexity, takes pictures in a time period/interval from t - n to t, where t is the current time.

The optical sensor 2 supplies (step S1) the images as a sequence of image frames, and a plurality of image frames are input as a sequence. The sequence can be viewed as an assembly/arrangement of temporally ordered images (which are 4 are indicated schematically). The sequence is input to the one or more detection and prediction units 11, 12 (step S2). First ver the detection unit 11 uses the sequence of image frames provided by the optical sensor 2 to detect objects of interest for the host vehicle running. The objects O1, O2 can contain pedestrians, other cars or trucks, obstacles, lane markings, lane boundaries, buildings and the like, and the detection unit 11 or a specific object information data model (which is stored in the detection unit 11 or in a separate sub-unit, which is stored in 2a not shown may be included) determines the following object information data. The object information data detected/determined by the detection unit 11 may include object boundary/demarcation frame coordinates.

The bounding box B can be viewed as a rectangular area/box placed around the detected object O1, O2 such that the detected object O1, O2 is precisely within the box B, which is preferably a two-dimensional box placed in the image frame is, fits. The bounding box coordinates may be the upper left and lower right corner coordinates that completely define the rectangular shape of the box B . In addition, the object information data can include the type of object, preferably the orientation of the object in relation to the coordinate system of the optical sensor of the host vehicle 1, a distance from the object to the optical sensor or to the origin of the coordinate system of the host vehicle, the dimensions or the size of the detected object and include its speed.

In a preferred alternative/modification, an identifier tracker unit 17 can also be included in the vehicle environment detection device 10, to which the information about the detected object 01, 02 can be sent in order to receive information about whether the detected object O1, O2 has already been detected in advance is or whether it is newly detected. This is described in more detail below.

The predictor model 12 also uses a trained neural network model (not shown) and applies time series analysis to the information of each detected object O1, O2 to derive/derive future object information data. The future object information data may include the bounding box coordinates, the one or more orientations, the distance, the one or more dimensions, and/or the velocity of each detected object in a future time interval extending from a current time t (or t + Dt) to a future time t + k included. The number of values for all future object information data can be predefined, e.g. B. be stored in the memory unit 7 or may be in accordance with the number of shorter time intervals between the time t + Dt and the time t + k.

With the detection and the prediction being performed, the host vehicle compensation unit 13 receives data (step S3) from the host vehicle running parameter unit 46 as well as a processing time lag determination unit 16 to calculate the position of the host vehicle 1 with respect to the time elapsing between the current time t to the the last image of the time interval between t - n and t has been captured (or processing has started) and the completion of the detection and prediction processing and the corresponding communication delays (latency) have elapsed. The processing time delay determination unit 16 can measure this time, which is denoted as “Dt”, it can estimate the time based on previous cases, or it can calculate the time when corresponding parameter data is available. The host vehicle running parameter unit 46 also provides the actual linear velocity and yaw of the host vehicle 1 at time t, based on which e.g. B. the new position of the host vehicle 1 at the time t + Dt can be compensated / corrected / determined by extrapolation.

The compensation of course has an effect on the distances detected by the detector unit and the predictor unit 11, 12 during their respective processing. Therefore, the values are preferably also balanced. In this regard, it should be repeated that the distances between the host vehicle 1 and any detected object 01, 02 can be determined by a separate AI/ML model, ie separate from the detector model or predictor model. The one or more separate models (not shown) may be sub-units of the detector model/predictor model; the models can be part of the device 10 that is carried out 2A is shown to be. The distance determination can be performed by the trained model based on the input of the image data provided by the optical sensor 2 .

After the host vehicle compensation has been performed and with the data from the detection unit and the prediction unit 11, 12, images provided in a sequence of image frames between time t (or time t+tD) and t+k can be output by the output unit 14 or the OCG generation unit 15 are generated (step S4), wherein the output unit 14 and the OCG generation unit 15 may be a single unit (which differs from the structure shown in 2a is shown, ver is divorced). The generation of the output, preferably a sequence of occupancy grids, in the time interval between time t+Dt and t+k is discussed in connection with 7 and 8th explained below.

The effect of the carrier vehicle compensation is also carried out by 5 , which calculates the relative distance/position of a detected object 01, 02 and the host vehicle 1 depending on the time, which is the current time t at the time the images were taken by the optical sensor 2 and the time t + Dt , after which the object detection and prediction processing has been completed, shows schematically described. In detail shows 5 the example case in which the host vehicle has a linear speed of 10 m/s and a parked passenger car 01 (whose speed is 0 m/s) is detected. However, at time t, since the processing of object detection, etc., the host vehicle 1 has passed the time Dt (which is in 5 shown as Δt) required to move the distance Δd from the original position. The compensation takes this and 5 shows the differences in relative distances/positions of the detected object O1 and the host vehicle 1.

Furthermore, an example flow of data/communication patterns between the various units, starting at the optical sensor 2 and providing images to the object detector model in the detection unit 11 and to a latency estimator included in the processing time delay determination unit 16, is through 6 shown. The detector model detects objects O1, O2 in the images and provides information about the detected objects to an identifier tracker unit 17, which also receives information about previously tracked objects from a storage unit (e.g. the unit 7 or 5). Based on the received information, the identifier tracker unit 17 determines whether the detected object O1, O2 has been previously detected and its information has been stored in memory. If so, the identifier tracker can forward the information about the detected objects O1, O2 and whether they were newly detected or not to the object predictor model in the prediction unit 12. The prediction unit 12 may forward information to the latency estimator and host vehicle compensation. The latency estimator can e.g. B. the time t when the images were input into the object detector model and the time when the object predictor model outputs the predicted/future information are received. The time difference can be used for the calculation/estimation of the latency/time delay that is input to the host vehicle offset.

In other words, the processing time delay determination unit 16 receives information from the current images at time t and the prediction model and provides an estimate of the delay time/latency to the host vehicle compensation unit 13, which also receives at least information about the linear velocity and the yaw rate of the host vehicle 1 at the current time t . The output of the host vehicle compensation unit 13 and the object prediction model unit 12 then contains future object information data in a time interval between time t+Dt and future time t+k. On this basis, as described above, a sequence of image frames can be generated in the future time interval between t(t+Dt) and t+k, which are preferably occupancy grid maps as further explained below.

An example of allocation generation is through 7 and 8th shown. 7 12 shows a possible image recorded by the optical sensor 2 of the host vehicle 1. The image shows a roadway, the unspecified surroundings of the roadway (which are listed in 8th also referred to as an “unknown area”) and another vehicle (object O1). After the object detection, a bounding box B is placed around the object O1 detected by the object detection and identified as “passenger car”.

Furthermore, in 7 a center of picture (picture frame) P1 and a center of bounding box B (marked as "P2") are also shown, as well as a virtual line crossing "P1" and a virtual line crossing "P1" and "P2 “ connects, where in 7 the first is labeled L1 and the second is labeled L2. The (polar) angle between the two lines L1, L2 is detected. The angle becomes in 7 identified as "Theta". With the bounding box B described earlier, the center of the bounding box shown in 2 is specified as "P2" and e.g. B. is located in the middle of the bottom of the bounding box B, and the center of the image frame "P1" (which is e.g. the center position of the bottom edge of the image frame if the optical sensor 2 is to be assumed to be at this location). the polar angle, which in 7 shown as "Theta" can be calculated and on this basis the location of the detected object O1, O2 in a Cartesian coordinate system can be calculated. In particular, the polar angle "Theta" and the distance from object O1 to the host vehicle can be used to calculate the location of object O1 in Cartesian coordinates.

Based on the information described above, i.e. including the picture frame, an occupancy grid (OCG) can be generated from the Cartesian coordinate system and the positions of the one or more objects O1, O2. Specifically, an occupied area of an OCG is calculated based on the (future) object information data, specifically, the orientation, the dimension, the distance to the host vehicle 1, and the bounding box B. Using the location of the detected object, its orientation and its dimension, the area occupied by this object O1, O2 in the occupancy grid map can be provided. In 8th is an example of transforming an image with the detected objects O1, O2 ( 8a) into an occupancy grid ( 8b) shown.

In 8a it can be seen that a truck and a passenger car were detected as the objects O1 and O2 running on the detected lane. The detection model has bounding boxes B around the two objects O1, O2 and based on the further (future) object information data discussed above, a transformation of the image shown in 8a is shown can be performed in such a way that an occupancy grid map shown in 8b shown can be generated. Each of the occupancy grids from time frame tD + t to time frame t + k shows areas occupied by detected objects O1, O2, preferably in a bird's eye view, as well as the roadway (which is marked as "free (i.e. unblocked) roadway space"), the Show surroundings of the lane (here marked as "unknown area") and optionally a part of the host vehicle.

The OCG sequence for the time interval between time t (t+Dt) and time t+k can be output to a display device in the host vehicle 1 such that the driver of the host vehicle 1 can use the information to adjust the travel route or the like. Additionally or alternatively, the output can be provided to a control unit of the (semi)autonomous host vehicle 1 such that the control unit can automatically control the travel route of the host vehicle 1 based on the information from the one or more OCG sequences.

In summary, a vehicle environment detection device 10 and a system 100 and a related computer program product are described that provide increased reliability and safety because, among other things, object information can be predicted in the future. This information can be used to avoid collisions during navigation with the host vehicle 1 or when the host vehicle 1 is an autonomous vehicle. Furthermore, the accuracy of the output data can be improved, among other things, due to the host vehicle position compensation.

An accurate occupancy grid map can be annotated with more object information output from the model, and a series of occupancy grid maps can be generated, with the occupied areas of the detected objects being represented statically or dynamically. The system can use only a mono camera or the like instead of multiple sensor sources as described in the prior art.

It should be noted that e.g. B. 1 and 2 show a possible example of the architecture of the device 10 and the system 100 described here and e.g. B. the distribution of subunits/units can be different. For example, some of the sub-units of the vehicle surroundings detection device 10 can be located in one or both of the computing units 8a, 8b or the entire vehicle surroundings detection device 10 can be located in one or more computing units 8a, 8b, while the data, in particular the output data for the control device 3 of the Carrier vehicle 1 are delivered directly for further processing and / or to a display device of the carrier vehicle 1.

As will be appreciated by those skilled in the art, the present disclosure described hereinabove and in the accompanying figures may be embodied as a method (e.g., a computer-implemented process, or any other process), an apparatus (which is a device, a machine , a system, a computer program product, and/or any other device) or a combination of the foregoing.

Aspects/examples of the present disclosure may be entirely software (including firmware, resident software, microcode, etc.) or a combination of software and hardware aspects that may be referred to as a "system". Furthermore, the present disclosure may take the form of a computer program product on a computer readable medium, including computer-executable program code embodied in the medium.

It should be noted that arrows may be used in drawings to represent communication, transmission, or other activity involving two or more entities. Double-headed arrows generally indicate that activity can occur in either direction (e.g., an instruction/requirement in one direction with a corresponding response back in the other direction, or peer-to-peer communications initiated by either entity), although in some situations activity in both directions need not necessarily occur.

Unidirectional arrows generally indicate activity exclusively or predominantly in one direction, although it should be noted that in certain situations such directional activity may actually involve activity in both directions (e.g., a message from a sender to a receiver and a confirmation back from the receiver to the sender or establishing a connection before transmission and terminating the connection after transmission). Thus, the type of arrow used in a particular drawing to represent a particular activity is exemplary and should not be considered limiting.

The present disclosure may be described with reference to flowchart illustrations and/or block diagrams of methods and devices, and with reference to a number of example views of a graphical user interface generated by the methods and/or devices. It will be understood that each block of the flowchart illustrations and/or block diagrams, and/or combinations of blocks in the flowchart illustrations and/or block diagrams, as well as the graphical user interface, can be implemented by computer-executable program code.

The computer-executable program code may be delivered to a processor of a general purpose computer, a special purpose computer, or another programmable computing device to create a particular machine such that the program code executed by the processor of the computer or other programmable computing device creates means for implementing the functions/acts/outputs specified in the flowchart, one or more block diagram blocks, figures and/or a written description.

The computer-executable program code may also be stored on a computer-readable memory that can instruct a computer or other programmable computing device to operate in a particular manner such that the program code stored in the computer-readable memory creates an article of manufacture that includes instruction means that implement the function/act/output specified in the flowchart, one or more block diagram blocks, figures, and/or a written description.

The computer-executable program code may also be loaded into a computer or other data processing device to cause a series of operational steps to be performed in the computer or other programmable device to create a computer-implemented process such that the program code running on the computer or executed in the further programmable device provides steps for implementing the functions/acts/outputs specified in the flowchart, block diagram one or more blocks, figures and/or written description. Alternatively, computer program implemented steps or acts may be combined with operator or human implemented steps or acts to carry out an embodiment of the disclosure.

It should be noted that terms such as For example, "server" and "processor" may be used herein to describe devices that may be used in certain aspects of the present disclosure and should not be construed to limit the present disclosure to any particular type of device unless the context permits not otherwise required. Thus, a device may include, without limitation, a bridge, router, bridge-router (brouter), switch, node, server, computer, device, or other types of device. Such devices typically include one or more network interfaces for communicating over a communications network and a processor (e.g., a microprocessor with memory and other peripherals and/or application-specific hardware) appropriately configured to perform device functions.

Communications networks may generally include public and/or private networks; may include local area networks, metropolitan area networks, metropolitan area networks, storage networks, and/or other types of networks and may employ communications technologies that include, but are not limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), network technologies, and internet technologies are.

It should also be noted that facilities use communication protocols and Messages (e.g., messages created, sent, received, stored, and/or processed by the device) and such messages can be transmitted through a communications network or medium.

Unless the context otherwise requires, the present disclosure should not be construed as limited to any particular communication message type, communication message format, or communication protocol. Thus, in general, without limitation, a communication message may include a frame, a packet, a datagram, a user datagram, a cell, or any other type of communication message.

Unless the context otherwise requires, references to particular communication protocols are exemplary, and it is understood that alternatives, as appropriate, include variations on such communication protocols (e.g., modifications or enhancements to the protocol that may be made from time to time). or other protocols that are either known or may be developed in the future.

It should also be noted that logic flows may be described herein to show various aspects of the disclosure and should not be construed to limit the present disclosure to any particular logic flow or logic implementation. The logic described may be divided into different logic blocks (e.g., programs, modules, functions, or subroutines) without changing the overall results or otherwise departing from the true scope of the disclosure.

Often, logic elements can be added, changed, omitted, performed in a different order, or implemented using different logic constructs (e.g., logic gates, loop primitives, conditional logic, and other logic constructs) without changing the overall results or otherwise departing from the scope of the disclosure to deviate

The present disclosure may be embodied in many different forms, including a graphics processing unit, computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general purpose computer), programmable logic for use with a programmable logic device (e.g., a field programmable gate array (FPGA) or another PLD), discrete components, integrated circuitry (e.g., an application specific integrated circuit (ASIC)), or any other means, including any combination thereof , but not limited to. Computer program logic that implements some or all of the described functionality is typically implemented as a set of computer program instructions, converted into computer-executable form, stored as such on a computer-readable medium, and executed by a microprocessor under the control of an operating system . Hardware-based logic that implements some or all of the functionality described may be implemented using one or more suitably configured FPGAs.

Computer program logic that implements all or part of the functionality previously described herein may be embodied in various forms, including source code form, computer-executable form, and various intermediate forms (e.g., forms implemented by an assembler, a compiler, linker, or locator) include, but are not limited to.

Source code may include a set of computer program instructions written in any of a variety of programming languages (e.g., an object code, an assembly language, or a high-level language such as Fortran, Python, C, C++, JAVA, JavaScript, or HTML) for use with different operating systems or operating environments are implemented. The source code can define and use different data structures and communication messages. The source code may be in computer-executable form (e.g., by an interpreter), or the source code may be converted (e.g., by a translator, an assembler, or a compiler) into a computer-executable form.

Computer-executable program code for performing operations of embodiments of the present embodiment may be defined in an object-oriented, scripted or non-scripted programming language such as. B. Java, Perl, Smalltalk, C ++ or the like. However, computer program code for performing operations of aspects of the present disclosure may also be written in conventional procedural programming languages such as. B. the "C" programming language or similar programming languages can be written.

Computer program logic that implements all or part of the functionality previously described herein may be executed at different times on a single processor (e.g., temporally overlapping), or may be executed on multiple processors at the same time or at different times Executed and can run under a single operating system process/thread or under different operating system processes/threads.

Thus, the term "computer process" generally refers to the execution of a set of computer program instructions, regardless of whether different computer processes execute on the same or different processors, and regardless of whether different computer processes execute under the same operating system process/thread or different operating system processes/threads are running.

The computer program may be embodied in any form (e.g., source code form, computer-executable form, or an intermediate form), either permanent or transient on a tangible storage medium such as a computer drive. a semiconductor memory device (e.g. RAM, ROM, PROM, EEPROM or flash-programmable RAM), a magnetic memory device (e.g. floppy disk or hard disk), an optical memory device (e.g .a CD-ROM), a PC card (e.g. a PCMCIA card) or other storage device.

The computer program may be embodied in any form in a signal transmitted using any of a variety of communication technologies, including but not limited to analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), network technologies, and Internet technologies are, can be sent to a computer.

The computer program may be distributed in any form as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), preloaded into or from a computer system (e.g., in a system ROM or hard disk). Server or an electronic bulletin board over the communication system (z. B. the Internet or the global network) are distributed.

Hardware logic (including programmable logic for use with a programmable logic device) that implements all or part of the functionality previously described herein can be designed using traditional manual methods or can be designed electronically using various tools such as . B. computer aided design (CAD), a hardware description language (e.g. VHDL or AHDL) or a PLD programming language (e.g. PALASM, ABEL or CUPL) can be designed, recorded, simulated or documented.

Any suitable computer-readable medium can be used. The computer-readable medium can e.g. B. but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or medium.

More specific examples of the computer-readable medium include an electrical connection having one or more wires, or another tangible storage medium such as a memory card. a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), compact disk read-only memory (CD-ROM). ) or other optical or magnetic storage device, but not limited thereto.

Programmable logic can be either permanent or transient in a tangible storage medium such as a memory card. a semiconductor memory device (e.g. RAM, ROM, PROM, EEPROM or programmable flash RAM), a magnetic memory device (e.g. floppy disk or hard disk), an optical memory device (e.g. a CD-ROM) or another storage device.

The programmable logic may be embodied in a signal communicated using any of a variety of communications technologies, including but not limited to analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), network technologies, and internet technologies can be sent to a computer.

The programmable logic may be distributed as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), preloaded into a computer system (e.g., in a system ROM or hard drive), or downloaded from a server or a electronic bulletin board via the communication system (e.g. the Internet or the worldwide network). Of course, some embodiments of the disclosure may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other aspects of the present disclosure are implemented entirely in hardware or entirely in software.

While certain exemplary aspects have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of the broad disclosure and are not limiting, and that aspects of the present disclosure are not limited to the specific constructions and arrangements shown and described , are limited as various other modifications, combinations, omissions, changes and substitutions are possible in addition to those set forth in the paragraphs described above.

Those skilled in the art will appreciate that various adaptations, modifications, and/or combinations of the aspects and examples just described can be configured. Therefore, it is to be understood that within the scope of the appended claims, the disclosure may be practiced otherwise than as specifically described herein. For example, unless expressly stated otherwise, the steps of processes described herein may be performed in different orders than those described herein, and one or more steps may be combined, divided, or concurrent be performed. Those skilled in the art will also appreciate that, given the disclosure, various aspects or examples of the disclosure described herein can be combined to form further aspects or examples of the disclosure.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US 2019/0317519 A1 [0004]
• WO 2019/177562 A1 [0005]

Claims (14)

Vehicle environment detection device for a carrier vehicle (1), which contains an optical sensor (2) which is communicatively connected to the vehicle environment detection device (10), the vehicle environment detection device (10) being configured for receiving image data from the optical sensor (2) mounted on the host vehicle (1) at a current time t; detecting one or more objects (O1, O2) in the image data; determining at least the distance to the host vehicle (1) and the speed of the one or more detected objects (O1, O2) at time t; Predicting at least one future distance to the host vehicle (1) of the one or more detected objects (O1, O2) in a predefined time interval from time t to a future time t+k; Compensating for the position of the host vehicle (1) based on a processing time lag Dt at time t+Dt and Outputting a sequence of image data containing at least the future distance between the host vehicle (1) and the one or more detected objects (O1, O2) in the interval from time t+Dt to time t+k. device after claim 1 comprising: - a detection unit (11) communicatively connected to at least the optical sensor (2) of the host vehicle (1) and a prediction unit (12); - the prediction unit (12) being communicatively connected to at least the detection unit (11) and a host vehicle balancing unit (13); - the host vehicle compensation unit (13) communicatively connected to at least a processing time delay determination unit (16), a host vehicle running parameter unit (46), the prediction unit (12) and an output image data unit (14); and - the output image data unit (14); wherein the detection unit (11) is configured to receive the image data from the optical sensor (2) at a current time t, to detect one or more objects (O1, O2) in the image data and object information data about the one or more detected objects ( O1, O2) to the prediction unit (12), wherein the object information data of each of the one or more detected objects (O1, O2) contain at least the distance to the host vehicle (1) and the speed at the current time t; the prediction unit (12) is configured to predict, at least on the basis of the object information data received from the detection unit (11), future object information data about the one or more detected objects (O1, O2), the future object information data being each of the one or the plurality of detected objects (O1, O2) contain at least future distance values to the host vehicle (1) between the current time t and a future time t+k; the host vehicle compensation unit (13) is configured to determine the position of the host vehicle (1) at a time t + Dt based on host vehicle running data received from the host vehicle running parameter unit (46) and the processing time lag Dt received from the processing time lag determination unit (16). was to correct; and the output image data unit (14) is configured, the corrected position of the host vehicle (1) at time t + Dt, the object information data output by the detection unit (11), and the future object information data output by the prediction unit (12). , and to generate a sequence of image data in a time interval from time t+Dt to time t+k. Apparatus according to at least one of the preceding claims, wherein the image data received by the detection unit (11) at the current time t contains a sequence of image frames containing a plurality of frames from a past time t - n to the current time t. Apparatus according to at least one of the preceding claims, wherein the object detection uses a trained machine learning model and/or a trained artificial intelligence model for detecting an object and the machine learning model and/or the artificial intelligence model is trained as the Object information data for each detected object (O1, O2) the type of the detected object (O1, O2), the orientation of the detected object (O1, O2) with respect to a host vehicle-based coordinate system, a bounding box of the detected object (O1, O2) and the Coordinates in the carrier vehicle-based coordinate system to determine the dimensions of the detected object (O1, O2), the distance to the carrier vehicle (1) and the speed of the detected object (O1, O2). Apparatus according to at least one of the preceding claims, wherein the prediction uses a trained machine learning model and/or a trained artificial intelligence model to perform a time series analysis to predict the future information data and the machine learning model and/or the artificial intelligence model is trained as the future object information data for each detected object (O1, O2) and for a sequence of image frames in a time interval between the current time t+Dt and a future time t + k the orientation of the detected object (O1, O2) with respect to a host vehicle-based coordinate system, a bounding box of the detected object (O1, O2) and the coordinates in the host vehicle-based coordinate system, the dimensions of the detected object (O1, O2), the distance to the carrier vehicle (1) and to determine the speed of the detected object (O1, O2). Device according to at least one of the preceding Claims 4 until 5 , wherein the model for machine learning and/or the model for artificial intelligence contains a convolutional neural network for detection and a recurrent neural network for prediction. Device according to at least one of the preceding claims, wherein the detection of the distance of each detected object (O1, O2) to the host vehicle (1) at a current time t is performed by a separate step that uses a model for machine learning and/or a model for uses artificial intelligence trained to determine the distance from the input image data received from the optical sensor (2) of the host vehicle (1) at the current time t, the received input image data being a sequence of image frames with frames from a time t - n at the current time t. Apparatus according to at least one of the preceding claims, wherein the data related to object detection and prediction and the compensated host vehicle driving data are merged in each image frame of a sequence of image frames in a time interval from time t+Dt to future time t+k and each image of each image frame in the time interval from time t+Dt to future time t+k is transformed into an occupancy grid of the surroundings of the host vehicle (1). Apparatus according to at least one of the preceding claims, wherein a sequence of occupancy grid image frames is output in a time interval from time t+Dt to future time t+k. Apparatus according to at least one of the preceding claims, wherein the compensation receives information about the processing delay time Dt caused by the detection and the prediction processing and the communication delay, and about the actual speed and the yaw rate of the host vehicle (1) at the time t and a correct position of the host vehicle (1) is determined at a time t+Dt. Apparatus according to at least one of the preceding claims, further comprising an identifier tracker unit (17) communicatively connected to a storage unit (5, 7) storing data of previously detected objects (O1, O2), the identifier tracker unit (17) configuring is to determine for each of the detected objects (O1, O2) whether the object is a newly detected object or a previously detected object, and the object prediction is configured to obtain from the identifier tracker unit (17) information about the detected object (O1, O2 ) that is a newly detected object or a tracked previously detected object. Device according to at least one of the preceding claims, wherein the device is also a computer program product comprising computer-readable instructions executable in a computer system containing one or more computing units (3, 8a, 8b). Device according to at least one of the preceding claims, wherein the units of the vehicle environment detection device 10 and a control device (4) of the host vehicle (1) are connected in terms of communication via hardware and/or software signal communication channels and the computing unit (3) of the host vehicle (1) and remote computing units (8a, 8b) are connected by communication via one or more networks (6). System containing an optical sensor (2) and a carrier vehicle (1) with a vehicle environment detection device (10) according to at least one of the preceding claims.

Images