DE102019206231A1 - Camera-based position determination for an object in the surroundings of a vehicle using an epipolar image line - Google Patents

Abstract

Method for the camera-based position determination for an object (62) in an environment (60) of an off-road vehicle, comprising obtaining a first image (24) and a second image (34), the first image (24) from a first camera from a Color camera (20) and a thermal camera (30) when capturing the environment (60) of the off-road vehicle is generated, the second image (34) from a second camera from the color camera (20) and the thermal camera (30) when capturing the Environment (60) of the off-road vehicle is generated; Defining an image area (342) in the second image (34) which at least partially comprises the object (62); Determining an epipolar image line (344) corresponding to a first image point (244) of the object (62) in the first image (24) in the second image (34); Determining an overlap portion (346) between the epipolar image line (344) and the image area (342); Determining depth information of the first image point (244) based on the overlap section (346).

Description

TECHNICAL AREA

The present invention relates to the field of off-road vehicles, in particular off-road vehicles that drive autonomously or partially autonomously. Specifically, the present invention relates to a device, a method, a computer program product, a computer-readable storage medium and a data carrier signal for the camera-based position determination for objects located in the vicinity of the off-road vehicle.

TECHNICAL BACKGROUND

Object recognition plays a central role in determining the position of objects that are in the environment of an autonomously or partially autonomously driving off-road vehicle. Off-road vehicles, in particular industrial vehicles, port vehicles, construction vehicles, agricultural machinery, robots and / or driverless transport systems (AGV), are confronted with certain environmental characteristics due to their specific areas of application. For example, traffic on a construction site is not regulated by road traffic infrastructures such as traffic lights and right-of-way signs. For this reason, it is necessary that construction vehicles, either with the help of the driver or an automated vehicle control system, correctly recognize the objects in the vicinity or moving objects, such as people, other vehicles and / or buildings, in order to avoid collisions and ensure a smooth flow of traffic guarantee.

Numerous approaches to a solution have been proposed in the past for the purpose of more reliable and faster object recognition. For example disclosed US 2016/0026880 A1 a driver assistance system for a vehicle comprising a broadband camera for generating an image with four light channels which are each assigned to a wavelength. WO 2011/160672 discloses a method for determining drivable path areas in order to assist a driver of a vehicle, the method comprising determining at least one current gesture of the vehicle by means of a localization system, obtaining a map model of the route by means of at least one image acquisition device attached to the vehicle.

However, there is a need to improve the speed and accuracy of object recognition. The known approaches to a solution are inadequate in that, after the false positives have been eliminated, a great deal of information is filtered out of the data obtained by means of environmental sensors without being used. The usability of the sensor data is therefore limited. The degree of confidence that detected sensor signals can be assigned to a certain object or a certain object class is also limited in the known approaches due to the type and manner of data processing and the physical conditions underlying the sensor system used.

The present invention is therefore based on the object of improving the position determination for objects located in the vicinity of a completely or partially autonomously driving vehicle with regard to speed and reliability.

The object is achieved by a device, a method, a computer program product, a computer-readable storage medium and a data carrier signal according to the independent claims.

The device according to the invention is used for the camera-based position determination for an object which is located in the surroundings of an off-road vehicle. The device according to the invention can be, for example, an electronic control or regulating unit (ECU = Electronic Control Unit), an electronic control or regulating module (ECM = Electronic Control Module) or a control / regulating unit for autonomous driving (e.g. a "Autopilot") act. The device can be located on the off-road vehicle, or outside or partially outside the vehicle. For example, the device according to the invention can be part of a central monitoring device in road traffic, in particular in city traffic, in motorway traffic, on an at least partially closed industrial, commercial site and / or on an agricultural area. The communication between the device according to the invention and the off-road vehicle or between the central monitoring device and the off-road vehicle can be wireless, for example via Bluetooth, infrared, near-field communication (NFC), radio, internet, intranet, cloud systems , Blockchain systems, 4G communication networks, 5G communication networks and / or wired systems.

The device comprises a data input for receiving a first image and a second image. The first image is generated by a first camera from a color camera and a thermal camera when capturing the surroundings of the off-road vehicle. The color camera is, for example, an RGB camera, in particular a camera with an RGB-based charge-coulpled density (CCD) chip or a complementary metal oxide semiconductor (CMOS) chip. The thermal camera is, for example, an infrared camera. The second picture is of one Second camera generated from the color camera and the thermal camera when capturing the surroundings of the off-road vehicle.

This means that the first image can be generated by the color camera, the second image being generated by the thermal camera. Alternatively, the first image can be generated by the thermal camera, the second image being generated by the color camera.

The device also includes an evaluation unit for defining an image area in the second image. This represents a sub-area of the second image which at least partially encloses the image of the object in the second image. Such an image area is referred to as a “bounding box” in the English specialist literature. The image area can be obtained in various ways: for example, it can be determined by means of object recognition based on the sensor data from the color or thermal camera. The object recognition is preferably based on semantic segmentation or model-based segmentation.

Alternatively, the image area can be determined based on another image area which represents a sub-area of the first image that at least partially encloses the image of the object in the first image. For the purpose of differentiation, the image area in the first image is a first image area, the image area in the second image being a second image area. The first image area can be determined by means of object recognition based on the sensor data from the thermal or color camera (or from the respective camera other than the camera from which the second image is generated). Semantic segmentation and / or model-based segmentation also come into consideration here.

For example, the first image area is modified based on a relative position between the color camera and the thermal camera, or between the image coordinate system of the color camera and the image coordinate system of the thermal camera. If the two image coordinate systems are the same (relative position equal to zero), the image of the object in the first and second image has the same image coordinates. In this case, the first image area can be used as the second image area, so that the second image area occupies the same image elements (e.g. pixels) in the second image as the first image area in the first image. Depending on the relative position, the second area can be obtained by a corresponding modification (e.g. rotation and / or displacement) of the first image area. The modified first image area obtained in this way can additionally be enlarged. The second image area generated based on the first image area advantageously ensures that image points in the first image area are also contained in the second image area.

The evaluation unit is also designed to define a first image point of the object in the first image. Any point on the image of the object in the first image can be selected for this. The selection takes place automatically or manually (e.g. by user input via an input interface). For the first image point determined in this way, an epipolar image line is determined in the second image by the evaluation unit.

The epipolar line is preferably determined as follows: a first conceptual connecting line between the first image point and a first focal point of the first camera is determined. Then several points are selected on the first line of thought. Furthermore, several second mental connecting lines are determined between the several points on the first mental connecting line and a second focal point of the second camera. These several second mental connecting lines intersect the image area in which the second image lies in several intersection points. These intersection points ultimately form the epipolar line. To determine the epipolar line, a straight line for fitting the multiple intersection points can be determined approximately.

The evaluation unit is also designed to determine an overlap section between the epipolar image line and the (second) image area. Depth information of the first image point can be determined based on the overlap section. The depth information designates a spatial coordinate outside the image area of the first camera and is a measure of the distance of the object relative to the camera.

According to the invention, the number of image points in the second image which is used to determine the depth information of the image point and thus to determine the position of the object can thus be greatly reduced. This advantageously leads to a reduction in the computational effort required to determine the position. The position determination for the autonomously driving off-road vehicle is therefore faster and maintains its accuracy and reliability at the same time.

Advantageous refinements and developments are specified in the subclaims.

According to one embodiment, the method according to the invention further comprises determining a second pixel in the overlap section, wherein the second pixel corresponds to the first pixel.

The second image point is preferably used to determine the object point of the object corresponding to the first image point by means of a triangulation based on the first image point and the second image point and the relative position between the first and second image point. This enables a particularly precise position determination.

According to a further embodiment, the second pixel is determined based on a correlation value between the first pixel on the one hand and a plurality of pixels of the overlap section on the other hand.

The second image point is determined from a plurality of image points located in the overlapping section in the second image: first of all, the correlation value is determined for each of the several image points located in the overlapping section. The pixel with the highest correlation value is then selected as the second pixel. This enables a particularly precise determination of the second image point, so that the position determination is particularly reliable.

According to a further embodiment, the correlation value relates to a correlation with regard to a gradient of the plurality of pixels of the overlap section.

For example, a first gradient value can be determined for the first image point and a second gradient value can be determined for each of the multiple image points located in the overlap section. From a comparison between the first gradient value and the respective second gradient value, that pixel of the overlapping section whose gradient value has the smallest difference to the gradient value of the first pixel can be determined. “Gradient” can be understood to mean the slope of an image contour (e.g. boundary between the image points of the object and the image points of the background) in the first image point or in the respective image points located in the overlap section.

According to a further embodiment, the first image point comprises an edge point of the object in the first image area.

The edge point lies on a contour of the object, e.g. in the boundary between the image points of the object and the image points of the background. In this way, particularly distinctive image points can be available for the selection of the first image point, so that the first image point can be determined in a simple manner.

The computer program product according to the invention is designed to be loaded into a memory of a computer and comprises software code sections with which the method steps of the method according to the invention are carried out when the computer program product is running on the computer.

A program belongs to the software of a data processing system, for example an evaluation device or a computer. Software is a collective term for programs and associated data. The complement to software is hardware. Hardware describes the mechanical and electronic alignment of a data processing system. A computer is an evaluation device.

Computer program products generally comprise a sequence of instructions which, when the program is loaded, cause the hardware to carry out a specific method that leads to a specific result. When the program in question is used on a computer, the computer program product produces the inventive technical effect described above.

The computer program product according to the invention is platform independent. That means it can run on any computing platform. The computer program product is preferably executed on an evaluation device according to the invention for detecting the surroundings of the vehicle.

The software code sections are written in any programming language, for example in Python, Java, JavaScript, C, C ++, C #, Matlab, LabView, Objective C.

The computer-readable storage medium is, for example, an electronic, magnetic, optical or magneto-optical storage medium.

The data carrier signal is a signal which the computer program product from a storage medium on which the computer program product is stored to another entity, for example another storage medium, a server, a cloud system, a wireless communication system 4G / 5G or a data processing facility, transmits.

• 1 eine schematische Darstellung einer Vorrichtung zur Positionsbestimmung gemäß einer Ausführungsform; und
• 2 eine schematische Darstellung des Prinzips der Positionsbestimmung mittels der Vorrichtung aus 1.

Embodiments will now be described by way of example and with reference to the accompanying drawings. Show it:

• 1 a schematic representation of a device for determining position according to an embodiment; and
• 2 a schematic representation of the principle of position determination by means of the device 1 .

In the figures, the same reference symbols relate to the same or functionally similar reference parts. The relevant reference parts are identified in the individual figures.

1 shows a schematic representation of a device 10 for the camera-based position determination for an object 62 which is in an environment 60 an off-road vehicle. The principle of position determination by means of the device 10 is based on 2 explained.

The device 10 includes a data input 12th to get a first image 24 and a second image 34 . The first picture 24 is from a first camera, here for example a color camera 20th when capturing the environment 60 of the off-road vehicle. The color camera 20th is for example an RGB camera, in particular a camera with an RGB-based Charge-Coulpled Density (CCD) chip or a Complementary Metal Oxide Semiconductor (CMOS) chip. The second picture 34 is from a second camera, here for example a thermal camera 30th when capturing the environment 60 of the off-road vehicle. The thermal camera 30th is for example an infrared camera.

Alternatively, the first picture 24 from the thermal camera 30th be generated, the second image 34 from the color camera 20th can be generated.

The device also includes an evaluation unit 16 to define a second image area 342 In the second picture 34 . This represents a part of the second image 34 which at least partially depicts the object 62 in the second picture encloses. Such an image area 342 is referred to as the “bounding box” in the English specialist literature. The second image area 342 can be obtained in different ways: for example, it can be obtained by means of object recognition based on the sensor data of the thermal camera 30th to be determined. The object recognition is preferably based on semantic segmentation or model-based segmentation.

Alternatively, as in 2 shown schematically, the second image area 342 based on a first image area 242 be determined, which is a partial area of the first image 24 represents that at least partially depicts the object 62 in the first picture 24 encloses. The first image area 242 can by means of object recognition based on the sensor data of the color camera 20th to be determined. Semantic segmentation and / or model-based segmentation also come into consideration here.

The evaluation unit 16 is also designed to have a first pixel 244 of the property 62 in the first picture 24 define. Any point on the image of the object can be used for this 62 in the first picture 24 be selected. The selection is made automatically or manually (e.g. by user input via an input interface).

To the first pixel determined in this way 244 becomes an epipolar image line 344 In the second picture 34 through the evaluation unit 16 determined. The epipolar line 344 is preferably determined as follows: first, a first conceptual connecting line is established between the first image point 244 and a first focal point of the first camera or the color camera 20th certainly. Then several points are selected on the first line of thought. Furthermore, several second mental connecting lines between the several points on the first mental connecting line and a second focal point of the second camera (or the thermal camera 30th ) certainly. These several second mental connecting lines intersect the picture area in which the second picture 34 lies in several points of intersection. Ideally, these points of intersection lie on a straight line that forms the epipolar line 344 Are defined. Therefore, it can be approximated to determine the epipolar line 344 a straight line for fitting the multiple points of intersection can be determined.

The evaluation unit 16 is also designed to have an overlap portion 346 between the epipolar image line 344 and the (second) image area 342 to determine. Based on the overlap section 346 depth information of the first pixel can be determined. The depth information designates a spatial coordinate outside the image area of the first camera (or the color camera 20th ) and is a measure of the distance to the object 62 relative to the color camera 20th .

The depth information can preferably be determined by the image points that are in the overlap section 346 in the two picture 34 are included with the first pixel 242 in the first picture 24 can be compared with regard to an image property (e.g. gradient of a contour). That pixel of the overlap section 346 that goes to the first pixel 242 is closest with regard to this image property, is selected as the second pixel. From the first pixel 242 , the second image point and the relative position of the two image coordinate systems or the two cameras 20th , 30th can use triangulation to obtain the depth information of the first pixel 242 to be determined.

List of reference symbols

10

contraption

12

first data entry

14th

second data input

16

Evaluation unit

18th

Signal output

20th

Color camera

24

first picture

242

first image area

244

first pixel

30th

Thermal camera

34

second picture

342

second image area

344

epipolar line

346

Overlap section

60

environment

62

Object (human)

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 2016/0026880 A1 [0003]
• WO 2011/160672 [0003]

Claims (11)

A method for the camera-based position determination for an object (62) in an environment (60) of an off-road vehicle, comprising: - Obtaining a first image (24) and a second image (34), the first image (24) from a first camera comprising a color camera (20) and a thermal camera (30) when capturing the surroundings (60) of the off-road vehicle is generated, the second image (34) being generated by a second camera from the color camera (20) and the thermal camera (30) when capturing the surroundings (60) of the off-road vehicle; - Defining an image area (342) in the second image (34) which at least partially comprises the object (62); - Determining an epipolar image line (344) corresponding to a first image point (244) of the object (62) in the first image (24) in the second image (34); - determining an overlap section (346) between the epipolar image line (344) and the image area (342); - Determination of depth information of the first image point (244) based on the overlap section (346). Procedure according to Claim 1 , further comprising determining a second pixel in the overlap portion (346), the second pixel corresponding to the first pixel (244). Procedure according to Claim 2 wherein the second pixel is determined based on a correlation value between the first pixel (244) on the one hand and a plurality of pixels of the overlapping section (346) on the other hand. Procedure according to Claim 3 wherein the correlation value relates to a correlation in terms of a gradient of the plurality of pixels of the overlap portion (346). Method according to one of the preceding claims, wherein the first image point (244) comprises an edge point of the object (62) in the first image (24). Method according to one of the Claims 2 to 5 , wherein the depth information is determined by means of triangulation using the first and second image points (244). Device (10) for camera-based position determination for an object (62) in an environment (60) of an off-road vehicle, comprising: - A data input (12, 14) for receiving a first image (24) and a second image (34), the first image (24) from a first camera comprising a color camera (20) and a thermal camera (30) when capturing the Environment (60) of the off-road vehicle is generated, the second image (34) being generated by a second camera from the color camera (20) and the thermal camera (30) when capturing the environment (60) of the off-road vehicle; - An evaluation unit (16) for defining an image area (342) in the second image (34) which at least partially comprises the object (62); wherein the evaluation unit (16) is designed to determine an epipolar image line (344) corresponding to a first image point (244) of the object (62) in the first image (24) in the second image (34); - The evaluation unit (16) being designed to determine an overlap section (346) between the epipolar image line (344) and the image area (342); - The evaluation unit (16) being designed to determine depth information of the first image point (244) based on the overlapping section (346). Use of the device (10) after Claim 7 in an off-road vehicle. Computer program product, comprising instructions which, when the program is executed by a computer, cause the computer to perform the method according to Claim 1 execute. Computer-readable storage medium on which the computer program product is based Claim 9 is stored. Data carrier signal that the computer program product according to Claim 9 transmits.

Images