DE102020111471A1 - Method and system for image recognition for an automated vehicle - Google Patents

Abstract

The present disclosure relates to a method for image recognition for an automated driving vehicle, comprising the acquisition of image data, the image data including a reference object and an object to be measured; determining a standard dimension of the reference object; and determining a dimension of the object to be measured based on the standard dimension of the reference object.

Description

The present disclosure relates to a method for image recognition for an automatically driving vehicle, a storage medium for carrying out the method, an image recognition system, a driver assistance system for automated driving of a vehicle with such an image recognition system and a vehicle with a driver assistance system for automated driving. The present disclosure relates in particular to an efficient size estimation of objects by means of dimensional comparison with standardized objects as they drive past.

State of the art

Driver assistance systems for automated driving are becoming increasingly important. Automated driving can be done with different degrees of automation. Exemplary degrees of automation are assisted, partially automated, highly automated or fully automated driving. These degrees of automation were defined by the Federal Highway Research Institute (BASt) (see BASt publication “Research compact”, edition 11/2012). For example, the vehicles with level 4 are fully autonomous in city operation.

The driver assistance system for automated driving uses sensors that perceive the environment on a visual basis, both in the area that is visible and invisible to humans. The sensors can be, for example, a camera, a radar and / or a LiDAR. In addition to high-precision maps, these are the main signal sources for driver assistance systems for automated driving. In reality, the vehicle environment consists of many different objects such as different vehicle types, people and animals, static objects such as standard doors and windows, garbage cans, traffic control systems, natural objects and terrain. It is therefore essential for automated driving to ensure precise perception of the environment.

Disclosure of the invention

It is an object of the present disclosure to specify a method for image recognition for an automatically driving vehicle, a storage medium for carrying out the method, an image recognition system, a driver assistance system for automated driving of a vehicle with such an image recognition system and a vehicle with a driver assistance system for automated driving, which enable a precise perception of the environment. In particular, it is an object of the present disclosure to reliably estimate a size of objects.

This problem is solved by the subject matter of the independent claims. Advantageous refinements are given in the subclaims.

According to an independent aspect of the present disclosure, a method for image recognition for an automatically driving vehicle, in particular a motor vehicle, is specified. The method comprises acquiring image data, the image data comprising a reference object and an object to be measured; determining a standard dimension of the reference object; and determining a dimension of the object to be measured based on the standard dimension of the reference object.

According to the invention, objects are dimensioned based on standard dimensions of a reference object located in the same image. This is done in particular by putting the object and the reference object in relation to one another. In this way, on the basis of a size comparison of the object with reference objects, for example a height of a person can be estimated by comparison with a standard dimension of a standard door while the person walks past it. This makes it possible to provide an efficient size estimation and thus an efficient calculation system for automated driving.

The term “object” as used in the context of the present disclosure includes dynamic and static objects. In particular, the term “object” includes people, vehicles, road construction properties, etc. However, the present disclosure is not limited to this and the term “object” can be any object that can or should be dimensioned in the context of automated driving.

The reference object is an object whose dimensions are known. In particular, there can be a database with a large number of reference objects and their associated dimensions. In some embodiments, the database can be an authority database. For example, global standard dimensions can be accessed from country-specific standard databases for reference objects.

The database can be contained in a central unit (backend / server) with which the vehicle is in communication. Alternatively, the database can be contained in the vehicle.

The image data can be recorded by means of an environment sensor system of the vehicle. The environment sensor system can include, for example, one or more cameras that the Capture image data or corresponding images. The environmental sensors of the vehicle can be set up to capture the image data while driving past the object or reference object. In some embodiments, the environment sensor system can capture the image data laterally with respect to a direction of travel of the vehicle. In other words, the viewing direction can be essentially perpendicular to the direction of travel and / or perpendicular to an object contact area in order to be able to capture an object that is as distortion-free as possible.

The image data are preferably two-dimensional image data. This is advantageous in comparison to a three-dimensional measurement, e.g. using a point cloud, since extensive equipment does not have to be used to evaluate a large number of three-dimensional data points in a time-consuming manner.

A dimension of the object to be measured is preferably determined using projective geometry. In particular, image processing and calculation can be carried out using projective geometry (e.g. with homogeneous coordinates) and, if necessary, taking additional data into account, e.g. from distances between objects and a known reference object. In this way, an efficient size estimation can take place in particular by means of projective geometry from two-dimensional image data, and thus a more efficient calculation system can be provided for the automated driving of the vehicle.

Preferably, determining a dimension of the object to be measured further comprises determining a perspective distorted dimension of the object to be measured; and rectifying the perspective distorted dimension to obtain the dimension of the object to be measured. This can be done using projective geometry, for example.

The term “perspective distorted dimension” refers to the fact that the camera of the vehicle records the image of the object and the reference object from a certain perspective, so that the three-dimensional object or reference object is shown distorted in the two-dimensional image. For example, if the camera is angled (below the horizontal: bird's eye view; above the horizontal: frog's eye view), the perspective shows the object either larger (or longer / higher / wider) or smaller (or shorter / lower / narrower). By rectifying by means of the projective geometry, the real dimensions of the object can be determined.

Images are preferably recorded essentially directly at the level of the object and at the level of the reference object in order to avoid distortion about the vertical axis.

For example, a two-dimensional measurement (e.g. length and / or width comparison) of the object to be measured and the reference object can first be carried out from a perspective distorted camera image. The calculation of the real object height ("rectification") can then be carried out by projective geometry with the addition of the real reference parameter from the database and / or compared with a second image of the reference object, which was recorded by the vehicle as undistorted as possible in the form of a 2D image . Such a reference image only coincides with the image of the object to be measured if the object is located directly in front of the reference object and the camera is optimally positioned with the least possible distortion. In all other cases, at least two images can be used in which the object and the reference object are recorded as free of distortion as possible.

• - einer perspektivisch verzerrten Abmessung des Referenzobjekts; und/oder
• - der perspektivisch verzerrten Abmessung des zu bemessenden Objekts; und/oder
• - der Standardabmessung des Referenzobjekts; und/oder
• - einem Winkel eines Bildsensors bzw. einer Kamera zu einer Horizontalen; und/oder
• - einem Abstand zwischen dem Bildsensor bzw. der Kamera und dem zu bemessenden Objekt; und/oder
• - einem kürzesten Abstand zwischen dem Bildsensor bzw. der Kamera und dem Referenzobj ekt.

Preferably, the distorted perspective dimension is rectified using at least one parameter selected from the group that includes or consists of the following:

• - a perspective distorted dimension of the reference object; and or
• - the perspective distorted dimension of the object to be measured; and or
• - the standard dimensions of the reference object; and or
• - An angle of an image sensor or a camera to a horizontal; and or
• a distance between the image sensor or the camera and the object to be measured; and or
• - A shortest distance between the image sensor or the camera and the reference object.

The object to be measured is preferably a person, wherein determining a dimension of the object to be measured further comprises determining a posture of the person, in particular based on a speed of the person, and determining a real size of the person taking into account the posture. The height of humans and animals varies with their speed. Thus, in addition to a measured height, e.g. the lunge can be analyzed, for example to determine the height of a person at rest.

According to a further independent aspect of the present disclosure, a software (SW) program is specified. The SW program can be set up to be executed on one or more processors, and thereby to execute the method described in this document for image recognition for an automatically driving vehicle.

According to a further independent aspect of the present disclosure, a storage medium is specified. The storage medium can comprise a SW program which is set up to be executed on one or more processors and thereby to execute the method described in this document for image recognition for an automatically driving vehicle.

According to a further independent aspect of the present disclosure, an image recognition system is specified. The image recognition system comprises one or more processors which are set up to thereby execute the method described in this document for image recognition for an automated vehicle, and in particular a motor vehicle.

The image recognition system is preferably implemented entirely in the vehicle. As an alternative to this, the image recognition system is partially implemented in the vehicle (in particular the environmental sensor system for capturing the image data) and partially implemented by a central unit.

The central unit can be a server, for example a backend of a vehicle manufacturer, which is in communication with the vehicle. The communication link between the vehicle and the central unit can take place via a mobile network, such as an LTE network or 5G network, but is not limited to this.

In some embodiments, the central unit can provide at least some of the functionalities of the image recognition system according to the invention. In particular, the central unit can include the database with the large number of reference objects and their associated dimensions.

According to a further independent aspect of the present disclosure, a driver assistance system for automated driving of a vehicle is specified. The driver assistance system comprises the image recognition system for a vehicle described in this document.

According to a further independent aspect of the present disclosure, a vehicle, in particular a motor vehicle, is specified. The vehicle comprises the driver assistance system for automated driving of a vehicle according to the embodiments of the present disclosure.

The term vehicle includes cars, trucks, buses, mobile homes, motorcycles, etc., which are used to transport people, goods, etc. In particular, the term includes motor vehicles for passenger transport.

The vehicle preferably includes an environment sensor system which is set up to record environment data. The environmental sensor system preferably comprises at least one LiDAR system and / or at least one radar system and / or at least one camera and / or at least one ultrasound system. The environment sensor system can provide the environment data (also referred to as “environment data”), which depict an area around the vehicle.

With automated driving, the longitudinal and / or lateral guidance of the vehicle takes place automatically. The driver assistance system therefore takes over the control of the vehicle. For this purpose, the driver assistance system controls the drive, the transmission, the service brake and the steering. To plan and carry out automated driving, information on the surroundings from the surroundings sensors that monitor the surroundings of the vehicle is received by the driver assistance system.

In the context of the document, the term “automated driving” can be understood to mean driving with automated longitudinal or lateral guidance or autonomous driving with automated longitudinal and lateral guidance. The automated driving can be, for example, driving on the motorway for a longer period of time or driving for a limited time as part of parking or maneuvering. The term “automated driving” includes automated driving with any degree of automation. Exemplary degrees of automation are assisted, partially automated, highly automated or fully automated driving. These degrees of automation were defined by the Federal Highway Research Institute (BASt) (see BASt publication “Research compact”, edition 11/2012).

With assisted driving, the driver continuously performs the longitudinal or lateral guidance, while the system takes on the other function within certain limits. With partially automated driving (TAF) the system takes over the longitudinal and lateral guidance for a certain period of time and / or in specific situations, whereby the driver has to constantly monitor the system as with assisted driving. With highly automated driving (HAF), the system takes over the longitudinal and lateral guidance for a certain period of time without the driver having to permanently monitor the system; however, the driver must be able to take control of the vehicle within a certain period of time. With fully automated driving (VAF), the system can automatically cope with driving in all situations for a specific application; a driver is no longer required for this application.

The four degrees of automation mentioned above correspond to SAE levels 1 to 4 of the SAE J3016 standard (SAE - Society of Automotive Engineering). For example, highly automated driving (HAF) Level 3 corresponds to the SAE J3016 standard. In addition, SAE J3016 provides SAE level 5 as the highest degree of automation, which is not included in the definition of BASt. SAE level 5 corresponds to driverless driving, in which the system can automatically cope with all situations like a human driver during the entire journey; a driver is generally no longer required.

Figure list

• 1 ein Flussdiagramm eines Verfahrens zur Bilderkennung für ein automatisiert fahrendes Fahrzeug gemäß Ausführungsformen der vorliegenden Offenbarung,
• 2A eine Draufsicht auf ein zu bemessendes Objekt und ein Referenzobjekt gemäß Ausführungsformen der vorliegenden Offenbarung,
• 2B eine Vogelperspektive eines zu bemessenden Objekts und eines Referenzobjekts gemäß Ausführungsformen der vorliegenden Offenbarung,
• 2C eine Seitenansicht eines zu bemessenden Objekts und eines Referenzobjekts gemäß Ausführungsformen der vorliegenden Offenbarung,
• 3 schematisch eine Körperhaltung einer Person und ihren Einfluss auf eine gemessene Höhe der Person gemäß Ausführungsformen der vorliegenden Offenbarung, und
• 4 schematisch ein Fahrassistenzsystem zum automatisierten Fahren gemäß Ausführungsformen der vorliegenden Offenbarung.

Exemplary embodiments of the disclosure are shown in the figures and are described in more detail below. Show it:

• 1 a flowchart of a method for image recognition for an automated driving vehicle according to embodiments of the present disclosure,
• 2A a top view of an object to be measured and a reference object according to embodiments of the present disclosure,
• 2 B a bird's eye view of an object to be measured and a reference object according to embodiments of the present disclosure,
• 2C a side view of an object to be measured and a reference object according to embodiments of the present disclosure,
• 3 schematically a posture of a person and its influence on a measured height of the person according to embodiments of the present disclosure, and
• 4th schematically a driver assistance system for automated driving according to embodiments of the present disclosure.

Embodiments of the disclosure

Unless otherwise noted, the same reference symbols are used below for elements that are the same and have the same effect.

1 shows a flow diagram of a method 100 for image recognition for an automated driving vehicle according to embodiments of the present disclosure. The procedure 100 can be implemented by appropriate software that can be executed by one or more processors (e.g. a CPU).

The procedure 100 includes in the block 110 acquiring image data, the image data including a reference object and an object to be measured; in the block 120 determining a standard dimension of the reference object; and in the block 130 determining a dimension of the object to be measured based on the standard dimension of the reference object.

In some embodiments, the image data can represent a single image, or the image data can correspond to separate images in each case as perpendicular as possible to the spanned object surface (as free of distortion as possible).

In addition or as an alternative, the standard dimensions can also be determined in different ways. In one example, the captured image data can be measured directly to obtain the standard dimensions. In a further example, standardized dimensions can be called up e.g. from data servers.

In general, the three-dimensional object and the three-dimensional reference object are shown distorted in the two-dimensional image. For example, if the camera is angled (below the horizontal: bird's eye view; above the horizontal: frog's eye view), the perspective shows the object either larger (or longer / higher / wider) or smaller (or shorter / lower / narrower). The real dimensions of the object can be determined by rectifying using projective geometry.

For example, a two-dimensional measurement (for example length and / or width comparison) of the object to be measured and the reference object can first take place from a perspective distorted camera image. The calculation of the real object height ("rectification") can then be carried out using projective geometry with the addition of the real reference parameter from the database. This in the following in detail with reference to the 2A, B and C. explained.

2A shows a top view of an object to be measured MO and a reference object RO . 2 B shows a bird's eye view of the object to be measured MO and the reference object RO . 2C shows a side view of the object to be measured MO and the reference object RO . All three figures span a 2D plane of the 3D spatial coordinates.

In this example the object to be designed is MO a person and the reference object RO is a door with standard dimensions that can be called up from a database.

The camera 11 of the vehicle 10 takes the image data while driving past the object MO or reference object RO on. This can either be sequential (e.g. if the measuring object MO and the reference object RO overlap) or by a single exposure. One line of sight BR the camera 11 is lateral with respect to a direction of travel FR of the vehicle 10 . In other words, is the direction of view BR essentially perpendicular to the direction of travel FR of the vehicle 10 .

However, the present disclosure is not directed to the vertical line of sight BR limited. For example, the direction of view BR be essentially perpendicular to a standardized surface, such as a closed top side of a door or a top edge of a door strip on the frame, in order to be able to measure its actual length from the 2D reference image with as little distortion as possible. In this case, the camera can have a higher probability of being perpendicular to the roadway / direction of travel in order to actually capture an object that is aligned in the longitudinal direction to the roadway. In the fine adjustment, however, a camera can always move.

First the object MO and the reference object RO identified and distances will be determined. In particular, the image data are saved while driving past the object MO (with central axis MA ) and the reference object RO (with longitudinal door edge TK ) and the object MO and the reference object RO recognized by image recognition.

The object MO and the reference object RO are ideally at the same level. If the object MO and the reference object RO are not at the same level, the distance in the y-direction (see 2 B) to the camera vertical axis z are recorded. Otherwise it is necessary to measure a distance b in the x-direction (camera 11 to the vertical axis of the measuring object MO ) and a distance a sufficient (shortest path to the reference object RO ; here the outer edge of the door TK the door outer panel; Typically it is the shortest distance, but in other embodiments other distances to standardized lengths can be used as long as they are essentially free of distortion, e.g. the length of the top edge of the door frame instead of the length of the door edge and thus the distance between the camera and the top edge of the door frame.

The distances can be determined by means of the vehicle's environment sensors 10 can be detected, for example by a radar system, a lidar system, ultra-short wave, etc.

The image data are then evaluated. The camera takes 11 a special perspective from an angle α of the camera 11 depends on the horizontal, the height of which H in the vehicle 10 meets the horizon when looking straight ahead (vanishing point). The object is now evaluated by means of image analysis MO in its distorted height h2 certainly. The reference object RO by image evaluation in its distorted height h1 to be determined.

The actual height of the object h4 can then be derived from the measured heights h1 and h2 as well as the standard dimensions or reference height h3 of the reference object RO , the angle α and the distances a and b. The reference altitude h3 can be determined by comparing objects (e.g. Germany = ISO, object = outer door leaf) with the database.

3 shows schematically a posture of a person and its influence on a measured height of the person.

The height of humans and animals varies with their speed. Thus, in addition to a measured height, the lunge can be analyzed, for example, in order to determine the height of a person at rest. In the example of the 3 is the measured height on the left c1 shown at rest. In the middle is the measured height c2 shown while walking. On the right is the measured height c3 shown while running. It can be clearly seen that the measured height varies depending on the posture. By taking the posture into account, a more precise estimate of the height of the person can be made possible.

Additionally or alternatively, a speed of the person can be deduced by means of the environmental sensor system and an associated person height can be linked from the database. A person in the idle state can then in turn be inferred from the link.

4th shows schematically a vehicle 10 with a driver assistance system 400 for automated driving according to embodiments of the present disclosure.

With automated driving, the vehicle is guided longitudinally and laterally 10 automatically. The driver assistance system 400 So takes over the vehicle guidance. The driver assistance system controls this 400 the drive 20th , The gear 22nd , the (e.g. hydraulic) service brake 24 and the steering 26th via intermediate units not shown in this example.

To plan and implement automated driving, information about the surroundings is provided by an environmental sensor system that monitors the surroundings of the vehicle from the driver assistance system 400 accepted. In particular, the vehicle can have at least one environment sensor 12th which is set up to record environmental data indicating the vehicle surroundings. The at least one environmental sensor 12th can for example comprise one or more LiDAR systems, one or more radar systems and / or one or more cameras.

The cameras can capture the image data or corresponding images that are used for dimensioning objects in accordance with the embodiments of the present disclosure. The camera of the vehicle can be set up to capture the image data while driving past the object or reference object. In some embodiments, the environment sensor system can capture the image data laterally with respect to a direction of travel of the vehicle.

According to the invention, objects are dimensioned based on standard dimensions of a reference object that is located in the same image or in sequentially recorded images (if it is not possible for the reference and measurement object to overlap, then two images can be used, ie a determination is made from two images, i.e. one image for the measurement object and one for the reference object). This is done in particular by setting the object and the reference object in relation to one another. In this way, on the basis of a size comparison of the object with reference objects, for example a height of a person can be estimated by comparison with a standard dimension of a standard door while the person walks past it. This makes it possible to provide an efficient size estimation and thus an efficient calculation system for automated driving.

Although the invention has been illustrated and explained in more detail by preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by the person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that there is a multitude of possible variations. It is also clear that embodiments cited by way of example really only represent examples that are not to be interpreted in any way as a limitation, for example, of the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to specifically implement the exemplary embodiments, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without the To leave the scope of protection that is defined by the claims and their legal equivalents, such as further explanations in the description.

Claims (10)

A method (100) for image recognition for an automated driving vehicle (10), comprising: Acquiring (110) image data, the image data comprising a reference object (RO) and an object to be measured (MO); Determining (120) a standard dimension of the reference object (RO); and Determining (130) a dimension of the object to be measured (MO) based on the standard dimension of the reference object (RO). The method (100) according to Claim 1 wherein the image data is two-dimensional image data. The method (100) according to Claim 1 or 2 , wherein the determination of a dimension of the object to be measured (MO) takes place using projective geometry. The method (100) according to one of the Claims 1 until 3 wherein determining a dimension of the object (MO) to be measured further comprises: determining a perspective distorted dimension of the object (MO) to be measured; and rectifying the perspective distorted dimension to obtain the dimension of the object to be measured (MO). The method (100) according to Claim 4 wherein the rectifying of the perspective distorted dimension takes place using at least one of the following parameters: a perspective distorted dimension of the reference object (RO); and / or the perspective distorted dimension of the object to be measured (MO); and / or the standard dimensions of the reference object (RO); and / or an angle of an image sensor (11) to a horizontal; and / or a distance (b) between the image sensor (11) and the object to be measured (MO); and / or a shortest distance (a) between the image sensor (11) and the reference object (RO). The method (100) according to one of the Claims 1 until 5 wherein the object (MO) to be measured is a person, and wherein determining a dimension of the object (MO) to be measured further comprises: determining a posture of the person, in particular based on a speed of the person; and determining a real height of the person taking into account the posture. A storage medium, comprising a software program which is set up to be executed on one or more processors, and thereby the method (100) according to one of the Claims 1 until 6th to execute. An image recognition system comprising one or more processors which are set up to carry out the method (100) according to one of the Claims 1 until 6th to execute. Driver assistance system (400) for automated driving of a vehicle (10), comprising the image recognition system according to Claim 8 . Vehicle (10), in particular motor vehicle, comprising the driver assistance system (400) according to Claim 9 .

Images