WO2019072451A1

WO2019072451A1 - Method for processing images

Info

Publication number: WO2019072451A1
Application number: PCT/EP2018/073344
Authority: WO
Inventors: Agnes SWADZBA
Original assignee: Robert Bosch Gmbh
Priority date: 2017-10-13
Filing date: 2018-08-30
Publication date: 2019-04-18
Also published as: DE102017218405A1

Abstract

The invention relates to a method for processing images, comprising the steps of capturing an environment by means of at least two cameras (2) placed in different locations, in order to obtain at least a first image (3) and a second image (4), which show the environment from different perspectives, determining first image regions (5, 6, 7) in the first image (3) and second image regions (8, 9, 10) in the second image (4), which image the same predefined examination regions (11, 12, 13) of the environment, and identifying at least one pair of first image region (5, 6, 7) and second image region (8, 9, 10) that shows the same scene content.

Description

Description title

Process for processing images prior art

The present invention relates to a method of processing images. Moreover, the invention relates to an associated computer program product and a control device for carrying out such a method.

Camera systems are known in the art in which multiple cameras capture the same area of an environment to allow algorithmic scene interpretation. Such systems are often in

Integrated vehicles to create all-round visibility around the vehicle. Current algorithms use camera-captured images of an environment to generate image information and related information about the environment. If the system is used in vehicles, information about the vehicle itself can also be generated. The processing of the images can be based on individual images or a comparison of temporally and spatially differently recorded images. A prerequisite for the realization of functions on spatially displaced images, such as in particular stereo flow and / or brightness harmonization and

Color harmonization of two adjacent cameras of a surround-view system, is the selection of suitable image areas. The functions rely on the fact that these areas contain identical scene content if possible. In current systems, a static image area is manually set per camera. However, these static image areas have the problem that in many scenarios, by covering distant objects by nearby objects in parts or completely divergent scene contents are present. This leads in many scenarios to degradation of the algorithms and functions of the system. Disclosure of the invention

The method according to the invention makes it possible to recognize those image areas which represent the same scene content. Further processing of the images can thus be carried out based on these image areas, whereby known algorithms and functions can be used without the risk of degrading said algorithms and functions.

The inventive method comprises the following steps: First, a recording of an environment with at least two takes place at different

Places placed cameras to get at least a first image and a second image. The first image and the second image thus show the environment from different perspectives. Subsequently, a determination is made of first image areas in the first image and second image areas in the second image, which image the same predefined examination areas of the environment.

Finally, at least one pair of first image area and second image area representing the same scene content is recognized. Thus, in particular a set of possible, for subsequent algorithms interesting, examination areas in the environment of the cameras in absolute world coordinates defined. These examination areas thus represent specific areas directly in the surroundings. These concrete areas are then projected and / or converted into the image plane of the individual cameras. This means that corresponding sections are found both in the first image and in the second image, on which the respective examination region of the environment is depicted. In this way, the first image areas and the second image areas can be determined. However, the determination of the first and second image areas is not yet meaningful with regard to the question whether in the first image areas and the second

Image areas the same scene content is shown. Thus, for example, an obstacle can be arranged in front of one of the cameras that obscures objects behind it. This obstacle, however, may not be visible from the other camera, for example. Thus, in the last described step, it is recognized which pairs of the first image area and the second image area actually display the same scene content. In particular, two image areas indicate the same scene content when all the objects in the one

Image area are also visible in the other image area. It is thus possible to recognize areas with identical scene content in order to subsequent algorithms for further processing the images said pairs of first image area and second image area to apply, whereby said algorithms can be used very effectively and reliably. The dependent claims have preferred developments of the invention for

Content.

It is preferably provided that the first image area and the second image area are determined on the basis of an extrinsic calibration and / or intrinsic calibration of the cameras. Said calibration of the cameras determines which areas of the environment are displayed on soft areas of the images. Thus, it can also be determined easily and with little effort on the basis of said calibration, on soft image areas

corresponding examination areas of the environment are presented. In this way it is guaranteed that in all pictures always the same

Inspection area is used as a basis for the image areas, regardless of a viewing angle and / or projection properties of the receiving camera. By defining the examination areas directly in the environment, the application specification of the

Sensor specification decoupled. This means that the examination areas can be selected independently of the cameras. In particular, the examination areas can be selected such that they are optimal for a subsequent further processing of the images. At the same time, the method is applicable to any number of cameras as long as two cameras always display overlapping areas of the environment. Should be for

Different applications, different study areas are used, this is easy and little effort possible.

Advantageously, the above-described step of recognizing comprises the following sub-steps: First, first image areas and second

Grouped image areas to pairs of a respective first image area and a second image area, wherein each of the first image area and the respective second image area depict the same examination area of the environment.

Subsequently, each first image area and each image area of a pair is coded by means of a descriptor. Finally, a determination is made that a pair of first image area and second image area represent the same scene content if the associated descriptors have a predefined descriptor spacing sign and / or have a minimum descriptor spacing. The coding by means of descriptors transforms the respective image regions in a descriptor space. Within this descriptor space, the descriptor distance can be determined simply and with little effort. Thus, such pairs of first image area and second can be easily and effortlessly

Detect image areas that represent the same scene content. The descriptor is advantageously such a descriptor which can withstand changes of illumination in the image and distortions due to the

Projection properties of the cameras is invariant. Such descriptors are known in particular from the prior art.

It is particularly advantageous that the descriptor of each first

Image area and / or second image area based on gradient histograms or binary comparison operators with consideration of the projection change due to changes in viewing angle. Such descriptors are known, for example, as "SIFT," "SURF," or "MAIL," as known in the art These descriptors provide the advantage of describing image areas in an abstract manner, and these descriptions can then be used to refer to To determine whether they represent the same scene content or due to

Concealments diverge from each other. This abstract valuation strategy provides valuable input for many applications described below. In an advantageous embodiment of the invention, a creation of a

Illustration of the environment from a predefined perspective, in particular a bird's eye view. This image from the predefined perspective, in particular the bird's-eye view, is calculated from the first image and the second image. In this case, a harmonization of the first image and the second image is necessary. Such harmonization is

in particular, a color harmonization and / or brightness harmonization, which is to be done so that in the resulting image from the predefined perspective, in particular bird's eye view, no or almost no visible boundaries between the first image and the second image arise. Such harmonizations are known in particular from the prior art. It is also known that such harmonizations fail when applied to image areas that do not have the same scene content represent. Thus, it is preferably provided that the harmonization of the first image and the second image takes place on the basis of such pairs of the first image area and the second image area, which in the step of the recognition are the same

Scene contents were detected. Thus, harmonization of high quality can be achieved. In particular, for the application

"Image harmonization" always the same amount of identical

Research areas defined. The examination areas are thus those areas of the environment which are relevant for assembling the individual pictures to the bird's-eye view. However, depending on the scene, a different pair of first image area and second

Image areas that represent the same scene content. Thus, image harmonization is significantly improved because even individual objects present in one image area but not in another image area will result in a significant reduction in the quality of harmonization. In contrast to the prior art, in which a static image area has always been selected, dynamic detection of pairs from the first image area and the second image area with the same scene content always identifies those image areas that reliably represent the same scene content. In an advantageous development is also provided that a

Correspondent search of corresponding pixels on the first image and the second image is performed. Such a correspondence search is particularly necessary if image processing methods such as stereo flow, structure-from-motion, object tracking or the like should be performed. A major challenge for correspondence search is the

Dealing with occlusions, since no correspondence can be found in the image area in which occlusion occurs. Ideally, therefore, in such image areas, the correspondence search method is not even applied in order to avoid false correspondence results and

Spare computing resources. Thus, it is provided that advantageously the

Correspondence search is performed only in such pairs of first image area and second image area that were recognized in the step of recognizing as the same scene content performing. This efficient pre-processing of the images makes it possible to apply the correspondence search method only to suitable image areas. This helps in particular, the effort of

Postprocessing of the correspondence results, since in particular, no false correspondence due to occlusion must be filtered out.

Furthermore, it is provided that a plurality of examination areas are used, each with a known distance to the cameras. These

Examination areas thus represent a grid that has an increasing distance to the cameras. By means of the above-described recognition of pairs of the first image area and the second image area, which represent the same scene content, it is thus possible to estimate a range of vision of the cameras. For this purpose, it is determined which of the examination areas have identical scene content. Once this is not the case, it can be seen that there is an obstacle. Since the distances of the examination areas to the cameras are known, the distance to the obstacle can be estimated by the distance of that examination area which for the first time does not lead to image areas with identical scene content. The examination areas are advantageously parallel planes which have different distances from the cameras.

The invention also relates to a computer program configured to carry out the method as described above. In particular, the computer program is a computer program product comprising instructions that, when executed on a processor of a computing device, cause the processor to perform the steps of the method as described above. The processor of the computing device is in particular a processor, particularly advantageously one

Microprocessor, a control device, in particular a control device of a vehicle.

The invention also relates to a machine-readable storage medium. The computer program is stored on the storage medium as described above.

Finally, the invention relates to a control device. The controller is

in particular a control device of a vehicle. The controller is configured to perform the method as described above. In particular, that is

Controller set up to run the computer program as described above. Short description of the drawing

Hereinafter, embodiments of the invention will be described in detail with reference to the accompanying drawings. In the drawing: is a schematic representation of a vehicle with a control device according to an embodiment of the invention, a schematic representation of the flow of the method according to an embodiment of the invention, a schematic representation of descriptors of a first alternative, in the method according to the embodiment of the invention a schematic representation of descriptors of a second alternative, which are used in carrying out the method according to the exemplary embodiment, a schematic representation of examination areas according to a first alternative of the method according to the embodiment of the invention, a first schematic representation of images in the examination areas according to the first alternative of the method according to the embodiment of the invention,

Figure 7 is a second schematic representation of images in the

Examination areas according to the first alternative of the method according to the exemplary embodiment of the invention,

FIG. 8 shows a schematic illustration of examination areas according to a second alternative of the method according to the exemplary embodiment of the invention, and Figure 9 is a schematic representation of images in the

Research areas of the second alternative of the

Method according to the embodiment of the invention. Embodiments of the invention

Figure 1 shows schematically a vehicle 18 comprising a plurality of

Cameras 2 and a control unit 1. The control unit 1 is designed to capture images of the surroundings of the vehicle 18 by means of the camera 2. The controller 1 is for performing an image processing method according to an embodiment of the invention.

In the following FIGS. 2 to 9, reference is made by way of example to two cameras 2, to a camera 2 pointing to the front and to a right side of the vehicle 18. These two cameras 2 take partially overlapping images of the surroundings of the vehicle 18.

FIG. 2 schematically shows a sequence of the method according to FIG

Embodiment of the invention. In the process, 2 images 3, 4 of the surroundings are recorded by means of the cameras. Thus, a first image is 3 and a second

4, wherein the first image 3 and the second image 4 with

different cameras 2 were recorded. Examination areas 1 1, 12, 13 are defined in the surroundings (see FIG. 5), to which corresponding first image areas 5, 6, 7 and second image areas 8, 9, 10 can be found. However, as can be seen in particular from the second image 4, not all first image areas 5, 6, 7 and second image areas 8, 9, 10 have the same scene content, since they are partially obscured by obstacles. In particular, it can be seen that a first first image area 6 and a first second image area 8 have identical scene contents. On the other hand, a second first image area 6 and a second second image area 9 as well as a third first image area 7 and a third second image area 10 have different scene contents. To recognize this, descriptors 16 are used.

By means of the descriptors 16, coding takes place from the first image regions 5, 6, 7 and the second image regions 8, 9, 10 in a descriptor space 17. By this abstract description of the first image regions 5, 6, 7 and second image regions 8, 9, 10 , can be in a safe and easy way determine whether the respective image areas 5, 6, 7, 8, 9, 10 have identical scene contents.

If two descriptors 16 in the descriptor space 17 have a minimum descriptor spacing and / or a descriptor spacing that falls below a predefined limit value, then the associated image areas 5, 6, 7, 8, 9, 10 are recognized as representing the same scene content. In the example shown in FIG. 2, this is the case for the first first image area 5 and the first second image area 8.

Figures 3 and 4 illustrate different descriptors 16 as used in Figure 2. Thus, FIG. 3 shows the use of projection-dependent SIFT descriptors. FIG. 4 shows the use of

projection-dependent BRIEF descriptors 15. Instead of the SIFT descriptor 14, a SURF descriptor can also be used. A SIFT Descriptor 14 or a SURF Descriptor are descriptors based on gradient histograms. The BRIEF descriptor 15 is one such descriptor based on binary comparison operations. In any case, the descriptors 14, 15, 16 are invariant to changes in illumination in the image and distortions due to the projection characteristics of the cameras 2. It's just the projection change by a

Viewing angle change to take into account. Such descriptors 14, 15, 16 generate the advantage that thereby image areas 5, 6, 7, 8, 9, 10 can be described in an abstract manner, namely by the

Descriptor space 17. These descriptions may then be used to determine for area pairs whether they contain equal scene content or diverge due to coverage.

In particular, only those image regions 5, 6, 7, 8, 9, 10 are compared that represent the same examination region 11, 12, 13. This means that one

Descriptor distance only for the first first image area 5 and the first second image area 8, the second first image area 6 and the second second

Image area 9, as well as for the third first image area 7 and the third second image area 10 are determined. Only these pairs of image areas 5, 6, 7, 8, 9, 10 represent the same examination area 1 1, 12, 13, so that only these

Couples can contain identical scene content at all. FIGS. 5 to 7 show a first application of the method according to the exemplary embodiment of the invention. FIG. 5 shows how, with respect to the vehicle 18 and thus with respect to the cameras 2, a first one

Examination area 1 1, a second examination area 12 and a third examination area 13 are defined. These examination areas 1 1, 12, 13 are relevant areas that are required in order to generate a view from a virtual bird's eye view from the images of the cameras 2. For the

Generating the bird's eye view is necessary to harmonize the color and brightness of the images taken by the cameras 2. This is basically known from the prior art.

However, known harmonization procedures have great weaknesses if the areas to be harmonized do not represent the same content. Therefore, it is detected by means of the method according to the exemplary embodiment of the invention, which image areas 5, 6, 7, 8, 9, 10, which represent the examination areas 1, 12, 13, have the same scene content.

FIG. 6 shows a first example. It can be seen that in the first image 3, the first first image area 5 represents the first examination area 11. The second first image area 6 represents the second examination area 12.

Finally, the third first image area 7 constitutes the third examination area

The same applies to the second image 4. Here, a first second image region 8 represents the first examination region 11. A second second image region 9 represents the second examination region 12. Finally, a third second image region 10 represents the third examination region 13.

As described above, it is recognized that only the third first image area 7 and the third second image area 10 show identical scene content. Thus, a harmonization of the images 3, 4 of the cameras 2 based on the third first image area 7 and the third second image area 10 is performed. This ensures that the highest quality harmonization can be carried out.

Figure 7 shows the same example of another environment. Here it can be seen that only the first first image area 5 and the second first image area 8 have the same scene contents. Thus, a great advantage of the described method, which can be seen in the fact that such examination areas 1 1, 12, 13 are detected depending on the environment, in all images 3, 4 lead to the same scene content. The respective selected image area 5, 6, 7, 8, 9, 10 in the example shown in FIG. 6 is different from the example shown in FIG. If, as in the prior art, one would only specify a static image area, this would, in one or another example, lead to a considerable deterioration in the harmonization of the images 3, 4

Cameras 2 lead.

Figures 8 and 9 show a second application of the method according to the embodiment of the invention. Again, it is shown in FIG. 8 that different examination areas 1 1, 12, 13 with respect to FIG

Vehicle 18 and thus defined with respect to the cameras 2. In particular, it is provided that the examination areas 1 1, 12, 13 represent surfaces which are arranged parallel to one another and are enlarged with increasing distance from the vehicle 18. By way of example, only three are shown in FIG

Examination areas 1 1, 12, 13 shown, advantageously, more examination areas can be used, which have a larger area with increasing distance from the vehicle 18. All examination areas 1 1, 12, 13 have a known predefined distance to the vehicle 18 and thus to the cameras 2.

By means of such an application, it is possible to estimate a visibility of the cameras 2. In FIG. 9, in turn, the first image 3 and the second image 4 are shown. On the basis of the method described above, image areas 5, 6, 7, 8, 9, 10 can be seen again, which are the same

Have scene content. On the basis of this statement can be

Determine examination areas 1 1, 12, 13 where there is no obstacle. Based on the known distance of the examination areas 1 1, 12, 13 can thus be a distance of the obstacle to the vehicle 18 and thus to the cameras 2 determine such that the closest to the vehicle examination area 1 1, 12, 13, for no identical scene content of the associated first image area 5, 6, 7 and second image area 8, 9, 10 was detected, serves as a measure of the maximum visibility of the respective cameras 2. Thus, a visibility of the cameras 2 and / or a distance of the obstacle to the cameras 2 and / or to the vehicle 18 can be determined easily and with little effort.

Claims

claims

1 . A method of processing images comprising the steps

· Recording an environment with at least two different ones

Places placed cameras (2) to obtain at least a first image (3) and a second image (4), which show the environment from different perspectives,

Determining first image regions (5, 6, 7) in the first image (3) and of second image regions (8, 9, 10) in the second image (4), which have the same predefined examination regions (1 1, 12, 13) the

Map the environment, and

• recognizing at least one pair of the same scene content from a first image area (5, 6, 7) and a second image area (8, 9, 10).

2. The method according to claim 1, characterized in that each first

Image area (5, 6, 7) and every other image area (5, 6, 7) are determined on the basis of an extrinsic calibration and / or intrinsic calibration of the first image (3) or second image (4) receiving camera (2).

3. The method according to any one of the preceding claims, characterized

characterized in that the step of recognizing comprises the following substeps:

Grouping the first image areas (5, 6, 7) and second image areas (8,

9, 10) to pairs of first image area (5, 6, 7) and second

Image area (8, 9, 10), which depict the same examination area (11, 12, 13) of the surroundings,

• coding each first image area (5, 6, 7) and every second

Image area (8, 9, 10) of a pair by means of a descriptor (14, 15,

16), and

Specify that a pair of first image area (5, 6, 7) and second image area (8, 9, 10) represent the same scene content if the associated descriptors (14, 15, 16) have a predefined one

Descriptor distance below and / or a minimum

Deskriptorabstand exhibit. A method according to claim 3, characterized in that the descriptor (14, 15, 16) of each first image area (5, 6, 7) and / or second image area (8, 9, 10) on gradient histograms or binary comparison operators with consideration of the projection change by Viewing angle changes based.

Method according to one of the preceding claims, characterized

characterized in that an illustration of the environment from a

predefined perspective from the first image (3) and the second image (4), wherein a harmonization of the first image (3) and the second image (4) based on such pairs of first image area (5, 6, 7) and second Image area (8, 9, 10), which were recognized as representing the same scene content in the step of recognizing.

Method according to one of the preceding claims, characterized

characterized in that a correspondence search of corresponding pixels on the first image (3) and the second image (4) is carried out, the correspondence search being performed only in such pairs of first image region (5, 6, 7) and second image region (8, 9, 10) recognized as representing the same scene content in the step of recognizing.

Method according to one of the preceding claims, characterized

in that a multiplicity of examination areas (1 1, 12, 13) each having a known distance from the cameras (2) are used and are identified on the basis of such pairs of first image area (5, 6, 7) and second image area (8, 9, 10 ), which in the step of recognizing it as the same

Scene contents have been detected representing those examination areas (1 1, 12, 13) are determined, where an obstacle (19) is present, based on the predefined distance of the examination areas (1 1, 12, 13) of the cameras (2) Distance of the obstacle (19) to the cameras (2) to determine.

A computer program adapted to carry out the method of any one of the preceding claims. Machine-readable storage medium on which the computer program according to claim 8 is stored.

Control device (1) adapted to carry out the method according to one of claims 1 to 7.