EP3555801A1

EP3555801A1 - Method for creating a cluster of image regions for detecting a target object in an image, computer program product, camera system and motor vehicle

Info

Publication number: EP3555801A1
Application number: EP17805197.5A
Authority: EP
Inventors: Swaroop KAGGERE SHIVAMURTHY; Sunil Chandra
Original assignee: Connaught Electronics Ltd
Current assignee: Connaught Electronics Ltd
Priority date: 2016-12-13
Filing date: 2017-11-29
Publication date: 2019-10-23
Also published as: DE102016124123A1; WO2018108524A1

Abstract

The invention relates to a method for creating at least one cluster (C_H, C_M, C_L) of image regions of an image (A) taken from a surrounding area (20) of a motor vehicle (10), which cluster is usable for detecting at least one target object (18) in said surrounding area (20), the method comprising the following steps: identifying a given number N_initialof characteristic features in the image (A) and dividing the image (A) into a plurality of image regions (T1A, T1B, T1C, T2A,..., T3), determining at least one parameter of the characteristic features to each image region (T1A, T1B, T1C, T2A,..., T3); clustering adjacent image regions (T1A, T1B, T1C, T2A,..., T3) with characteristic features having their statistical parameter within a common parameter area. According to the invention the parameter of the characteristic features or one of the parameters of the characteristic features is a feature saliency of the characteristic features in a respective image region (T1A, T1B, T1C, T2A,..., T3). The invention further relates to a corresponding use of the method for detecting target objects (18), computer program product, corresponding camera system (12) and motor vehicle (10) with such kind of camera system (12).

Description

Method for creating a cluster of image regions for detecting a target object in an image, computer program product, camera system and motor vehicle

The invention relates to a method for creating at least one cluster of image regions of an image taken from a surrounding area of a motor vehicle, which cluster is usable for detecting at least one target object in said surrounding area. The method comprises the following steps: (i) identifying a given number of characteristic features in the image; (ii) dividing the image into a plurality of image regions; (iii) determining at least one statistical parameter of the characteristic features to each image region; and (iv) clustering adjacent image regions with their characteristic features having statistical parameters within a common parameter area. The invention further relates to a corresponding use of the method for detecting target objects, computer program product, a corresponding camera system and motor vehicle with such kind of camera system.

Camera systems for motor vehicles are already known from the prior art. In general, such a camera system includes at least one camera, which is attached to the motor vehicle and captures a surrounding area of the motor vehicle. Several such cameras can also be employed, which capture the entire surrounding of the motor vehicle. The camera mounted on the motor vehicle provides a temporal sequence of images of the surrounding area, namely a plurality of images per second. This image sequence is then communicated to an electronic evaluation device, which processes the captured images and is able to provide very different functionalities in the motor vehicle based on the images. Presently, the interest applies to the detection of target objects located in the depicted surrounding area of the vehicle. If a target object is detected in the images, thus, this target object can be tracked in the sequence of images. For this purpose, in the prior art, the optical flow method is usually used, in which characteristic features such as for example edges and/or corners are detected in the images and a flow vector is calculated to each characteristic feature, which specifies the direction of movement and the speed of movement of the characteristic feature in the sequence of images.

In the detection of target objects in the camera images, there is a great challenge in differentiating between characteristic features, which are associated with different target objects. In order to ensure this differentiation, usually, a so-called clustering is effected such that several regions with characteristic features are combined to a cluster, which represents a detected target object.

Document WO 2015/014882 A1 describes a method for detecting at least one target object in a surrounding area of a motor vehicle based on an image of the surrounding area, wherein a given number of characteristic features are identified in the image, at least one part of the image is divided into a plurality of image regions connected to image nodes, and a local feature density of the characteristic features is determined to each image node or image region respectively. The detection of the target object includes that several of the image nodes/image regions are combined to a cluster representing the target object depending on the respective feature density. The statistical parameter of the characteristic features used in this method is a feature density of the characteristic features in a corresponding image region.

The recognition of characteristic features out of an image taken by a camera can be very CPU intensive even if no characteristic features at all are present in some regions of the image.

It is an object of the invention to provide a method for creating at least one cluster of image regions for target detection, a use of this method for detecting target objects, a computer program product, a camera system as well as a motor vehicle improved over the prior art.

This object is achieved by a method, a use of said method, a computer program product, a camera system as well as a motor vehicle having the features according to the respective independent claims. Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.

According to the inventive method for creating at least one cluster of image regions of an image taken from a surrounding area of a motor vehicle, the parameter of the characteristic features or at least one of the parameters of the characteristic features is a feature saliency. Feature saliency of characteristic features in an image is a measure of feature relevance. This measure is known to the skilled person from different kind of analysis of image features for identification of important image regions of an image (keyword: saliency map). The saliency of a characteristic feature (or of a simple pixel) is based on its contrast with its surroundings.

The method makes sure that no characteristic features are missed on key objects due to a low statistical parameter with respect to high texture regions (i.e. regions with high local feature density of the characteristic features) in the image.

Each of the image regions is allocated to the cluster or one of the clusters by clustering image regions, wherein a score of the characteristic features within the image regions of each cluster is in a common parameter range (=parameter score, e.g. feature saliency score). These parameter ranges can form a predefined number X of cluster classes or cluster types. Based on three parameter ranges of the feature saliency score/ feature variance score, there will be three cluster types X = 3: one with a low feature variance score CL, one with a mid-size feature variance score C_M and one with a high feature variance score C_H. The image regions can be tiles or other rectangular image cells.

Preferably, another parameter of the characteristic features in an image region is a feature density of the characteristic features in the respective image region. In other words, the method comprises the steps of (a) determining both, the feature saliency and the feature density of the characteristic features to each image region of the image and (b) clustering adjacent image regions with their characteristic features having a common parameter range with respect to feature saliency score and/or feature density of the characteristic features.

According to a preferred embodiment of the invention, the image is divided into the plurality of image regions by means of an iterative dividing scheme allowing division of the image into of image regions of different size. The iterative dividing scheme uses intermediate image regions to end up with the final image regions the image is divided into. These final regions do not have uniform size. Because at least some of the final regions are relatively large, the method using this iterative scheme does not need a time-consuming and/or expensive merging procedure at the end.

According to another preferred embodiment of the invention, at least one of the parameters of the features in the respective region, especially the feature saliency, is determined to each region in each iterative step, wherein in each iterative step the size of possible sub-regions of the region generated by division in the previous iterative step is based on the parameter assigned to the corresponding region generated in the previous iterative step.

According to yet another preferred embodiment of the invention, the (final) image regions with highest feature density and the image regions with lowest feature density are combined to clusters separately.

The clustering process ends up with a number of clusters C_x of different cluster-types X. These clusters C are formed by thresholding against features density and feature sali- ency of the characteristic features. The number of characteristic features assign to each of the clusters C is N_features-

According to an embodiment of the method, a re-distribution step of re-distributing the characteristic features in each cluster (C) by

N initial

features _j

clusters

is provided. The re-distributions of the characteristic features in each cluster C is based on feature density and feature saliency of the characteristic features. N_Cx is the number of features assign for redistributions to cluster C_x', divided into X cluster-classes with i clusters for each of the classes 1 = 1 , 2, ..n_x.

The number of features allowed to re-distributes in each of C_x', with i = 1 , 2, .. n is:

Area(C )

c^Cx ∑Area(C_x' )

i=l,...,N

with i = 1 , n.

The invention further relates to the use of the aforementioned creating method for detecting at least one target object in said surrounding area of the motor vehicle. In other words, the invention further relates to an inventive method using the

aforementioned method for creating at least one cluster of image regions of an image taken from a surrounding area of a motor vehicle for detecting at least one target object in said surrounding area of the motor vehicle. The computer program product according to the invention comprises computer- executable program code portions having program code instructions configured to execute the aforementioned method.

The camera system for a motor vehicle according to the invention includes a camera for providing an image of a surrounding area of the motor vehicle and includes an evaluation device adapted to perform the aforementioned method.

Finally, the invention relates to a motor vehicle with the aforementioned camera system.

Further features of the invention are apparent from the claims, the figures and the description of figures. All of the features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations or else alone.

Now, the invention is explained in more detail based on a preferred embodiment as well as with reference to the attached drawings.

In the drawings:

Fig. 1 shows a motor vehicle including a camera system according to a

preferred embodiment of the invention;

Fig. 2 shows a first step of an iterative dividing scheme for dividing an image of a surrounding area of the vehicle into a plurality of image regions;

Fig. 3 shows a second step of an iterative dividing scheme for dividing an image of a surrounding area of the vehicle into a plurality of image regions;

Fig. 4 shows a third step of an iterative dividing scheme for dividing an image of a surrounding area of the vehicle into a plurality of image regions;

Fig. 5 shows a forth step of an iterative dividing scheme for dividing an image of a surrounding area of the vehicle into a plurality of image regions; Fig. 6 shows a fifth step of an iterative dividing scheme for dividing an image of a surrounding area of the vehicle into a plurality of image regions;

Fig. 7 shows a sixth step of an iterative dividing scheme for dividing an image of a surrounding area of the vehicle into a plurality of image regions;

Fig. 8 shows a final step of an iterative dividing scheme for dividing an image of a surrounding area of the vehicle into a plurality of image regions;

Fig. 9 shows a clustering step for clustering the final image regions shown in

Fig. 8;

Fig. 10 shows a flow chart of a corresponding method for detecting target objects according to a preferred embodiment of the invention.

In Fig. 1 , a motor vehicle 10 according to an embodiment is shown in schematic illustration. The motor vehicle 10 is for example a passenger car. It includes a camera system 12 having a camera 14. The motor vehicle 10 is on a road 16. On the road 16, there is additionally a target object 18, which can be an obstacle. The camera system 12 is for example a collision warning system and serves for warning the driver of the presence of the target object 18 in a surrounding area 20 of the motor vehicle 10. The camera system 12 thus serves for detecting the target object 18 and preferably also for tracking the target object 18 in the surrounding area 20. Within the camera system 12, images A are taken by means of the camera 14, which are then processed by means of an electronic evaluation device (signal processor) not illustrated in more detail. The evaluation device receives the images from the camera 14 and processes them. The evaluation device can be integrated in the camera 14 or it can be a component of the camera system 12 separate from the camera 14. Optionally, the images can also be displayed on a display in the motor vehicle 10 (not shown), wherein the detection of the target object 18 can for example be effected to the effect that the target object 18 is provided with a border in the images.

The camera 14 can be a CMOS camera or a CCD camera or any image capturing device, which is formed for detecting light in the visible spectral range. The camera 14 is preferably a video camera continuously providing a sequence of images. The electronic evaluation device then processes the image sequence in real time and can detect and track the target object 18 based on this image sequence.

In the following a method for creating clusters C of image regions of an image A taken from the surrounding area 20 of the motor vehicle 10 is described. These kind of clusters C can be used for detecting target objects 18 in the surrounding area 20. The following notations/initials are used:

Notations Meaning

ROI Region of interest

Nfeatures Number of features in each cluster

Ntiles Initial number of tiles/slices.

DensitVinitiai Initial feature density

Varianceinitiai Initial feature saliency/variance

Threshold^ minimum threshold to allow maximum number of features

Thresholdadaotive Maximum threshold value

TileSize_Xmin Minimum tile size along X— axis. This value is configurable.

TileSize_Ymin Minimum tile size along Y— axis. This value is configurable.

Fe turej_ensi_ty Feature density.

FeatU re_variance Feature saliency/variance

Xtiles Tiles size along X— axis or tile width

Ytiles Tiles size along Y— axis or tile height

Ninitial Predefine number of total features

Nciusters Total number of clusters formed as per their densities

Table 1 : Details of notations/initials used

The method for creating clusters C comprises three main steps:

(i) identifying/detecting a given number of characteristic features in the image A or a ROI (ROI: region of interest) in the image A;

(ii) dividing the image A or the ROI into a plurality of final image regions by use of an iterative dividing scheme in consideration of two statistical parameter of the

characteristic features to each image region, namely (a) the feature saliency (Feature_Variance) of the characteristic features in a respective image region and (b) the feature density (Feature_densit_y) of the characteristic features in a respective image region (shown in Figs. 2 to 8); and

(iii) clustering adjacent final image regions with their characteristic features having statistical parameters within a common parameter area (shown in Fig. 9).

Feature detection refers to methods that aim at computing abstractions of image information and making local decisions at every image point whether there is an image feature of a given type at that point or not.

Many computer vision algorithms use feature detection as the initial step, so as a result, a very large number of feature detectors have been developed like (harries, FAST, LOG, HOG etc.). When feature detection is computationally expensive and there is time constraints embedded application, a higher level algorithm may be used to guide the feature detection stage, so that features are distributed on desired areas in given region of interest.

Figs. 2 to 8 show seven steps of an iterative dividing scheme for dividing the image A into a plurality of final image regions T1 A, T1 B, T1 C, T2A, T3 as shown in Fig. 8. With this method, the image or ROI of the image is iteratively subdivided into a number of tiles or other kind of image regions based on the statistical parameter (a) feature saliencies score (feature variance score) and (b) feature density of the characteristic features within each tile. The tiles size is not fixed. A scene can, e.g., have blocks with multiple sized rectangular tiles. Only final minimum block size width and height is configurable, i.e. sub divided block width and height should be more than configured minimum block width and height. The feature saliencies score is a measure for a parameter area of similar feature saliencies.

Fig. 2 shows a first step of the iterative dividing scheme for dividing the image A into a plurality of image regions. In the following these image regions are called "tiles" T. The image A is divided into three (intermediate) tiles T1 , T2, T3 and local statistical analysis of feature density and feature saliency (spatial feature variance) is performed in each tile. Each tile has a characteristic feature saliency score fcs (more general: feature parameter score), fcs = 400 for T1 , fcs = 2000 for T2 and fcs = 200 for T3. Tiles having higher feature densities and higher feature saliencies variance (T2 in the shown example) are further subdivided equally in to smaller tiles.

This step is required to further differentiate the regions into smaller regions based on feature densities and feature saliencies. So that smaller regions will represents higher feature densities. And the regions which do not have higher features densities will clustered together and adapt saliency threshold as per tile sizes. The smaller tile size will have higher saliency threshold due to high feature density.

Fig. 3 shows the result of this subdivision. After subdivision local statistical analysis of feature densities and feature saliencies in each tile/slice will be performed. As a result, tile T2 is sub-divided into tiles T2A having a re-calculated feature saliency score fcs = 1200 and T2B having a re-calculated feature saliency score fcs = 800.

Fig. 4 shows a sub-divisions of tile T2A as per feature density and higher feature saliencies variance. Further smaller tiles are analysed to extracts the features densities and feature saliencies. Again these smaller tiles are subdivided based on their local statistical analysis of feature densities and feature saliencies. Figs. 5 to 7 show a further division of tiles as per feature densities (Feature_densit_y) and higher feature saliency

( FeatU re_variance)■

This iterative dividing scheme of subdividing smaller tiles will continue until the process reached to the minimum configurable tiles size as shown in Fig. 8.

After reaching this distribution with minimum configurable tiles size adjacent final image regions with their characteristic features having statistical parameters within a common parameter area are clustered to Clusters C of different cluster-classes/cluster-types.

Fig. 9 shows the result of the clustering of adjacent image regions with their

characteristic features having statistical parameters within a common parameter area. The tiles with highest densities and lowest density are clustered separately.

Finally re-distributions of the characteristic features in each cluster C based on feature density and feature saliency can be performed. In the example the image A (ROI) is arranged in three different types of clusters C_H , C_M , C_L (N_Ci_Uster = 3): 1 . Clusters C_H with high feature saliency;

2. Clusters C_M with mid(-sized) feature saliency; and

3. Clusters C_L with low feature saliency.

As in the normal cases, most of the features will be distributed inside the Cluster with high feature saliency. Since the overall aim of this invention is to improve feature distributions uniformly. For that we further selects the number of features in each cluster as per following equation (1 ):

N initial

features _j

clusters

As a result, the image A is finally divided into ten clusters C with three different feature saliencies scores (low, mid, high), namely four clusters C_L with a low saliency score, three clusters C_M with a mid-sized saliency score and three clusters C_H with a high saliency score.

This whole process is fully adaptive and automatic in nature and will insure the distributions of the features extracted/detected using various computer vision based feature detector approach will be uniform in nature.

The proposed method will make sure that features are not missed on key objects due to low saliency score with respect to high texture region in the scene.

Fig. 10 shows a corresponding flow chart of the method for efficient feature distributions based on feature clustering by analysing global and local statistical properties of scene. The method comprises the following step:

Step 1 (S1 ): Detect predefined number N of features in the image A or a ROI of the image.

Step 2 (S2): Subdivide the ROI (image A) into number of tiles/slices as shown in Fig. 2.

Step 3 (S3): Performing local statistical analysis of feature densities and feature saliencies in each tile/slice. Step 4 (S4): Find the tiles which have maximum value of Feature_densit_y and Feature_variance and sub-divide that tiles into further tiles according to the following rule:

IF (X≥ 2 x TilesSizex_mi_n OR Y≥ 2 x TilesSize_Ymin) continue with the following request:

IF (X,iies > Ytiies) : divide in half along X - axis or tile width OR

IF (X,iies < Ytiies) : divide in half along Y - axis or tile height OR

IF (X,iies == Ytiies) : divide in half along either ( X - axis OR Y - axis ) or divide both axis

ELSE: Move to the tile which have next maximum value of Feature_density and

FeatU re_variance■

Tiles having higher feature densities and higher feature saliencies variance, further subdivided equally in to smaller tiles. This step is required to further differentiate the regions into smaller regions based on feature densities and feature saliencies. So that smaller regions will represents higher feature densities. And the regions which do not have higher features densities will clustered together and adapt saliency threshold as per tile sizes. The smaller tile size will have higher saliency threshold due to high feature density.

Further smaller tiles are analysed to extracts the features densities and feature saliencies. Again these smaller tiles are subdivided based on their local statistical analysis of feature densities and feature saliencies.

This process of subdividing smaller tiles will continue until the process reached to the minimum configurable tiles size, which is requested in Request R ("all Tiles processed"). In case the process did not reach to the minimum configurable tiles size (path N), steps 3 and 4 will be performed again on the next iterative (sub-)level. Otherwise (path Y) the method will continue with Step 5. The steps S3, S4 and R are forming a loop of an iterative dividing scheme.

Step 5 (S5): Creating the clusters C

(a) Clustering tiles with low saliency score C_L' (with i = 1 , 2, 3 ...): The biggest tile will represent an image region with low feature density and variance. This tile will be considered as starting point of a cluster which has low saliency score. Neighbour tiles with similar low feature density and feature saliency (variance) will be merged to create a cluster C_L ¹ until all neighbour-tiles are merged. In the following another tile with similar feature density and variance to the cluster C_L ¹ has to be found. This tile is a starting point for another cluster C_L ² with low saliency score. Again, neighbour tiles with similar low feature density and variance will be merged to create cluster until all neighbour-tiles are merged. This process has to be repeated to create all low saliency score clusters C_L ^1"4.

(b) Clustering tiles with highest saliency score C_H' (with i = 1 , 2, ...)

The smallest tile will represent an image region with high feature density and variance. This till will be considered as a starting point of a cluster which has high saliency score. Start merging neighbour tiles with similar high feature density and variance to create cluster C_H ¹ until all neighbour-tiles are merged. In the following another tile with similar feature density and variance to the cluster C_H ¹ has to be found. This tile is a starting point for another cluster C_H ² with high saliency score. Again, neighbour tiles with similar high feature density and variance will be merged to create a cluster until all neighbour- tiles are merged. This process has to be repeated to create all high saliency score clusters C_H ^1"3.

(c) Clustering tiles with mid(-sized) saliency score C_M' (with i = 1 , 2, ...)

Remaining unlabeled tiles belongs to mid saliency clusters C_M. Find first unlabeled tile. This tile will be considered as start of cluster C_M ¹ with mid-sized saliency. Start merging neighbour tiles which are unlabeled to create cluster C_M ¹ until all neighbour-tiles are merged. In the following another tile with similar feature density and variance to the cluster C_M ¹ has to be found. This tile is a starting point for another cluster C_M ² with midsized saliency score. Again, neighbour tiles with similar sized feature density and variance will be merged to create a cluster until all neighbour-tiles are merged. This process has to be repeated to create all mid-sized saliency score clusters C_M ^1"3.

Step 6 (S6): Re-distribution of the characteristic features in each of the clusters: The clustering process ends up with a number of clusters C of different cluster-types C_X with X = H, L, M (C_H, C_M, C_L) in the example. These clusters are formed by thresholding against features density and features variance. The number of characteristic features assign to each cluster C is N_features-

Ncx is the number of features assign for redistributions to cluster C_X', divided into X cluster-classes with i clusters for each of the classes 1 = 1 , 2, ..n_x.

In the example given by Fig. 9, the number n of different clusters for the "low" type L is 4. The number of features allowed to re-distributes in each of C_L', with i = 1 , 2, .. n=4 is:

Area{C_L )

Area(C_L' )

The re-distribution procedure for clusters of the "high"- and the "mid-size"-types/classes is similar.

In other words: the re-distributions of the features is done by weighted area occupied by each part of a single class clusters. This is done because each of C_L', with i = 1 , 2, ... n, assign number of features as per area occupied by them.

List of Reference signs vehicle 10 camera system 12 camera 14 road 16 target object 18 surrounding area 20 image A image region T1 A, T1 B, T1C, T2A, T2B, ..., T3 cluster C, CH, CM, CI_

S1 -S6 steps

R request

Y yes

N no

E end

Claims

1 . A method for creating at least one cluster (C_H, C_M, C_L) of image regions of an image (A) taken from a surrounding area (20) of a motor vehicle (10), which cluster is usable for detecting at least one target object (18) in said surrounding area (20), the method comprising the following steps:

identifying a given number N_initiai of characteristic features in the image (A) and dividing the image (A) into a plurality of image regions (T1 A, T1 B, T1 C, T2A, T3),

determining at least one parameter of the characteristic features to each image region (T1 A, T1 B, T1 C, T2A, T3); and

clustering adjacent image regions (T1 A, T1 B, T1 C, T2A, T3) with characteristic features having their statistical parameter within a common parameter area, characterized in that

the parameter of the characteristic features or one of the parameters of the characteristic features is a feature saliency (fcs) of the characteristic features in a respective image region (T1 A, T1 B, T1 C, T2A, T3).

2. The method according to claim 1 ,

characterized in that another parameter of the characteristic features in an image region (T1 A, T1 B, T1 C, T2A, T3) is a feature density of the characteristic features in the respective image region (T1 A, T1 B, T1 C, T2A, T3).

3. The method according to claim 1 or 2,

characterized in that the image (A) is divided into the plurality of image regions (T1 A, T1 B, T1 C, T2A, T3) by means of an iterative dividing scheme (S4, R) allowing division of the image (A) into of image regions (T1 A, T1 B, T1 C, T2A, T3) of different size.

4. The method according to claim 3,

characterized that at least one of the parameters (fcs) of the features in the respective region, especially the feature saliency, is determined to each region in each iterative step, wherein in each iterative step the size of possible sub-regions of the region generated by division in the previous iterative step is based on the parameter (fcs) assigned to the corresponding region generated in the previous iterative step.

5. The method according to claim 3 or 4,

characterized in that the image regions (T1 A, T1 B, T1 C, T2A, T3) with highest feature density and the image regions (T1 A, T1 B, T1 C, T2A, T3) with lowest feature density are combined to clusters (C_H, C_M, C_L) separately.

6. The method according to any one of the preceding claims,

characterized by a re-distribution of the characteristic features in each cluster (C_H,

JV features = N initial I /N clusters '

7. A use of the method according to any one of the preceding claims for detecting at least one target object (18) in a surrounding area (20) of a motor vehicle (10).

8. A computer program product comprising computer-executable program code portions having program code instructions configured to execute the method according to one of claims 1 to 6.

9. A camera system (12) for a motor vehicle (10) including a camera (14) for

providing an image (A) of a surrounding area (20) of the motor vehicle (10) and including an evaluation device adapted to perform a method according to any one of the claims 1 to 7.

10. A motor vehicle (10) with a camera system (12) according to claim 10.