WO2015014873A1

WO2015014873A1 - Method for training an algorithm for recognizing predetermined patterns in image data, camera system and motor vehicle

Info

Publication number: WO2015014873A1
Application number: PCT/EP2014/066335
Authority: WO
Inventors: James Mcdonald; John Mcdonald
Original assignee: Connaught Electronics Ltd
Current assignee: Connaught Electronics Ltd
Priority date: 2013-07-31
Filing date: 2014-07-30
Publication date: 2015-02-05
Anticipated expiration: 2016-01-31
Also published as: DE102013012781A1

Abstract

The invention relates to a method for training an algorithm (18) for recognizing predetermined patterns in image data provided by means of a camera of a motor vehicle, including the steps of: a) providing (S1) a test database (17) with test image data, wherein the test image data includes an overall amount (G) of patterns; b) selecting (S2) an exemplary subset (S) of patterns, which are representative of predetermined objects; c) first training (S3) of the algorithm (18) based on the selected subset (S) of the patterns and herein providing a preliminarily trained algorithm (18); d) applying (S4) the preliminarily trained algorithm (18) to the entire test database (17) and herein providing a result list (19) of entries; e) discarding (S5) entries from the result list (19), in which the association of the recognized pattern with the object has been correctly performed by the preliminarily trained algorithm (18); and f) second training (S6) of the preliminarily trained algorithm (18) based on the remaining entries.

Description

Method for training an algorithm for recognizing predetermined patterns in image data, camera system and motor vehicle

The invention relates to a method for training an algorithm for recognizing predetermined patterns in image data provided by means of a camera of a motor vehicle. In addition, the invention relates to a camera system for a motor vehicle including a camera for providing images, as well as including an image processing device with an algorithm, which has been trained according to such a method. The invention also relates to a motor vehicle witch such a camera system.

Pattern recognition algorithms are already known from the prior art in various

configuration. Such pattern recognition algorithms serve for recognizing predetermined patterns in images as well as for classifying the recognized patterns. Such algorithms are also employed in camera systems of motor vehicles and then in particular serve for detecting target objects such as for example other vehicles, pedestrians and other potential obstacles. If such a target object is detected, thus, this object can then be tracked by means of the camera system. The camera system can serve as a collision warning system, by means of which the driver of the motor vehicle is warned of a possible collision with a target object.

All of the pattern recognition algorithms have in common that they have to be trained or taught with respect to the patterns, which are to be recognized, before the algorithm is then finally transferred to the camera system. In training a pattern recognition algorithm, usually, a test database is provided, which includes test image data, i.e. a very great number of images. This test image data includes an overall amount of patterns, which are to be "communicated" to the algorithm and which are then taken as a basis for the training. Thus, these patterns are to be "learned" by the algorithm. An already trained algorithm is then capable of recognizing the learned patterns as well as patterns, which vary from the learned patterns, but which have some similarity, in any images. Thus, if a pattern recognition algorithm is trained with respect to a plurality of patterns, thus, not only these learned patterns, but also patterns can then be detected, which are in a tolerance range of the learned patterns.

Usually, the learned patterns are each associated with a target object such that classification of the patterns can be performed. Therein, several patterns can be associated with each target object or each type of object (vehicle, pedestrian etc.). If a pattern is then detected in the operation of the camera system, which is associated with a certain target object, thus, the pattern recognition algorithm is not only able to recognize the pattern, but also provide it with a designation, i.e. determine, with which target object this pattern is associated (classification).

In the prior art, usually, both positive and negative patterns are defined. Positive patterns are those patterns, which are to be recognized and output by the algorithm, because these positive patterns are associated with target objects, which are to be detected in the operation of the camera system. In comparison, negative patterns are those patterns, which must not be recognized by the algorithm and are to be filtered out, respectively. Namely, these negative patterns are associated with target objects, which are irrelevant to a camera system of a motor vehicle and thus do not have to be detected.

Training a pattern recognition algorithm is associated with considerable effort in the prior art: As already explained, in the prior art, a very large database with test image data is provided, which includes a great number of patterns. In the prior art, this entire database is manually searched, and all of the required patterns are manually (for example on the PC) selected and associated with a certain target object and correspondingly designated. For this purpose, each frame is manually analyzed one after the other. Then, the patterns are divided in positive and negative patterns. The algorithm is then trained based on all of the patterns. Therein, known learning methods can be used for training the algorithm.

Such an approach requires considerable expenditure of time. Namely, corresponding personnel has to correctly extract and designate all of the patterns and determine, with which target object the respective patterns are associated, respectively. Thus, the entire test database has to be searched by the personnel. This presents an extremely tedious and methodic task and therefore may result in errors - even with best intentions. Such errors in the pattern recognition cannot be tolerated in particular in motor vehicles.

After the selection and the designation of all of the patterns have been completed, in the prior art, there either is no guarantee that the quality of the selected patterns is optimal for training the algorithm. Therefore, manual filtering of the patterns often has to be performed. This filtering only means that prima facie erroneous patterns, i.e. those patterns, which previously were manually incorrectly designated, are discarded or corrected. This presents a further tedious task and cannot guarantee in any way that the selected patterns taken as a basis for training the algorithm have sufficient diversity in order to train the algorithm to the effect that it is also able to detect similar patterns. Thus, the diversity of the patterns overall presents a very important parameter in training a pattern recognition algorithm, because in the nature very similar patterns also often occur, which are to be recognized by the algorithm. A further disadvantage is in that due to the plurality of training patterns, the training of the algorithm alone takes relatively much time. Namely, this time is proportional to the number of the training patterns. In some cases, the training can even take several weeks.

One object of the invention is in providing a method, by which a pattern recognition algorithm can be trained very precisely on the one hand and without much effort on the other hand. A further object is in providing a camera system as well as a motor vehicle.

These objects are solved by a method, by a camera system as well as by a motor vehicle having the features according to the respective independent claims. Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.

A method according to the invention serves for training an algorithm for recognizing predetermined patterns in image data provided by means of a camera of a motor vehicle. Presently, a learning operation is understood by the term "training", with which

predetermined patterns are "communicated" to the algorithm, which the algorithm then is to recognize in the operation of the camera system, wherein similar patterns being in a tolerance range are also to be recognized by the algorithm.

According to step a), a test database with test image data for training the algorithm is provided, wherein the test image data includes an overall amount of patterns. This test image data can for example include a plurality of test images, which have for example been captured by means of a camera. Thus, the test image data can be acquired by means of a camera.

According to step b), an exemplary subset of patterns is selected from the overall amount of the patterns (in particular manually, for example on a PC), which are representative of predetermined objects. Preferably, only a small subset of standard patterns is selected, namely preferably with a maximum possible diversity, as it is manually still possible. The selection of the subset of patterns can be performed depending on the concrete application. According to step c), then, first training of the algorithm is performed based on the selected subset of the patterns such that a preliminarily trained algorithm is provided.

According to step d), this preliminarily trained algorithm is applied to the entire test database. As a result, the preliminarily trained algorithm outputs a result list of entries, which each specify a pattern recognized by the preliminarily trained algorithm as well as a designation of an object associated with this pattern.

According to step e), the list of results is analyzed (manually or automatically), and entries are discarded from the list of results, in which the association of the recognized pattern with the object was correctly performed by the preliminarily trained algorithm. In other words, the correct entries are removed from the result list since these entries do not include any usable information about the possible error sources of the preliminarily trained algorithm and thus do not contribute to the further improvement of the algorithm.

According to step f), then, second training of the preliminarily trained algorithm is performed based on the remaining entries of the result list.

Compared to the prior art, the pattern recognition algorithm can therefore be trained with considerably reduced effort because not the overall amount of the patterns has to be manually extracted from the test image data and labeled, but only a subset of patterns, which are representative of predetermined target objects. This subset of patterns can for example only include less than 20%, in particular less than 10%, of the overall amount of the patterns. Then, the pattern recognition algorithm can be trained based on this selected subset of the patterns such that the algorithm is taught to recognize and correspondingly classify this subset of patterns as well as similar patterns. Because the diversity and thus the variety of the patterns within the selected subset is very high, the preliminarily trained algorithm is also capable of identifying a plurality of other similar patterns. The above mentioned list of results can therefore be analyzed without much effort after all of the correct entries have been removed from the list of results.

The list of results of the preliminarily trained algorithm preferably includes the following types of entries:

- correct entries, in which the positive patterns were correctly recognized and the association of the recognized pattern with the object was also correctly performed - these entries are removed from the result list because they do not contribute to the improvement of the algorithm; and/or

- erroneous entries, in which the recognized pattern was associated with an incorrect object by the preliminarily trained algorithm - these entries can then be taken as a basis for the second training in order to train the algorithm also with respect to these patterns; and/or

- erroneous entries, in which a pattern was recognized, which would not have been allowed to be recognized by the preliminarily trained algorithm - these false detections can for example be included in a negative database of the algorithm within the scope of the second training such that the detection of these patterns can be prevented in the future.

Optionally, based on the list of results, it can also be examined if patterns of the overall amount have not been recognized by the preliminarily trained algorithm, but which would have had to be recognized by the algorithm. Then, a third training of the algorithm can be performed based on the non-recognized patterns. This further improves the algorithm.

Optionally, it can also be provided that the above mentioned steps d) to f) are repeated until at least a preset percentage of the overall amount of the patterns was correctly recognized by the algorithm. In this manner, a particularly precise and reliable pattern recognition algorithm can be provided.

In addition, the invention relates to a camera system for a motor vehicle including a camera for providing images of an environmental region of the motor vehicle, and including an electronic image processing device with an algorithm, which has been trained according to the method according to the invention.

A motor vehicle according to the invention, in particular a passenger car, includes a camera system according to the invention.

Further features of the invention are apparent from the claims, the figures and the description of figures. All of the features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone are usable not only in the respectively specified combination, but also in other combinations or else alone. Now, the invention is explained in more detail based on a preferred embodiment as well as with reference to the attached drawings.

There show:

Fig. 1 in schematic illustration a motor vehicle with a camera system according to an embodiment of the invention; and

Fig. 2 a flow diagram of a method according to an embodiment of the invention.

A motor vehicle 1 illustrated in Fig. 1 is for example a passenger car. The motor vehicle 1 has a camera system 2 with for example four cameras 3, 4, 5, 6, which are disposed distributed on the motor vehicle 1 . The cameras 3, 4, 5, 6 for example overall capture the environment around the motor vehicle 1 . For example, the entire environment of the motor vehicle 1 and therefore a 360° image can be captured.

The camera 3 is a front camera disposed in the front area of the motor vehicle 1 , for example on a front bumper 7. The camera 3 is therefore disposed on a front of the motor vehicle 1 . The second camera 4 is for example a rearview camera, which is disposed in the rear area, for example on a rear bumper 8 or a tailgate. The lateral cameras 5, 6 can be integrated in the respective exterior mirrors.

The first camera 3 captures an environmental region 9 in front of the motor vehicle 1 . Correspondingly, the camera 4 captures an environmental region 10 behind the motor vehicle 1 . The lateral cameras 5, 6 each capture an environmental region 1 1 and 12, respectively, laterally besides the motor vehicle 1 . The cameras 3, 4, 5, 6 can for example be so-called fish-eye cameras having a relatively wide opening angle, which for example can be in a range of values from 160 ° to 200°. The cameras 3, 4, 5, 6 can be CCD cameras or CMOS cameras. They can also be video cameras, which each are able to provide a plurality of frames per second. These images are communicated to a central electronic image processing device 13, which processes the images of all of the cameras 3, 4, 5, 6.

Optionally, the image processing device 13 can be coupled to an optical display device 14, which is for example an LCD display. Then, very different views can be presented on the display 14, which can be selected according to driving situation. For example, the image processing device 13 can generate an overall presentation from the images of all of the cameras 3, 4, 5, 6, which shows the motor vehicle 1 and its environment 9, 10, 1 1 , 12 from a bird's eye view and thus from a point of view, which is located above the motor vehicle 1 . Such a "bird eye view" is already prior art and can be generated by image processing.

The image processing device 13 can also identify objects 15, which are located in the environment 9, 10, 1 1 , 12 of the motor vehicle 1 . In the embodiment according to Fig. 1 , a pedestrian 15 is shown as the object, who is identified in the images by means of the image processing device 13 and then is for example displayed as an obstacle to the driver. In order to detect the target object 15, so-called characteristic features can be extracted from the images, such as for example Harris points and/or FAST features and/or edges and/or corners. These characteristic features are then subjected to pattern recognition.

An algorithm is stored in a memory 16 of the image processing device 13, which is configured to recognize predetermined patterns in the images. However, before this algorithm was transferred to the image processing device 13 and stored in the memory 16, the algorithm had to be trained with respect to the patterns, which are to be

recognized in the operation of the camera system 2. Such a method serving for training such an algorithm is explained in more detail below with reference to Fig. 2:

In a first step S1 of the method, a test database 17 is provided, which includes test image data. This test image data is for example acquired by means of a camera. The test image data includes an overall amount G of patterns, for which the algorithm is to be trained. Thus, predetermined types of objects (pedestrians, vehicles and the like) can be imaged in the test image data each with a plurality of examples and from different perspectives. The overall amount G of patterns can therefore respectively include a plurality of possible patterns for each type of object.

In a further step S2, an exemplary subset S of patterns is selected from the overall amount G, which are representative of predetermined types of objects. For example, therein, several patterns for a pedestrian and/or several patterns for a vehicle can be selected. The subset S is selected such that the patterns are as different as possible and thus have maximum diversity as possible, as it is manually still possible. Thus, preferably, such patterns are selected for a certain type of object, which differ from each other at least to a preset extent.

In a further step S3, the algorithm is taught with the selected subset S. In this step S3, learning algorithms can be used as they are already known from the prior art. Herein, the subset S of the patterns it taught or "communicated" to the algorithm denoted by 18 in Fig. 2, as it is schematically indicated in Fig. 2.

In a further step S4, then, the preliminarily trained algorithm 18 is applied to the entire test database 17. This means that the entire test database 17 and thus all of the test image data is taken as a basis for the preliminarily trained algorithm 18. As a result of this image processing, a list of results 19 is output by the algorithm 18, which includes entries respectively specifying a detected pattern and a classification of the pattern (the associated object).

In a further step S5, all of the entries of the list of results 19 are sorted in the following groups:

- correct entries, in which the pattern has been correctly recognized and classified;

- erroneous entries, in which the recognized pattern has been associated with an

incorrect object;

- erroneous entries, in which a pattern was recognized, which would not have been allowed to be recognized.

According to step S5, the correct entries are removed from the list of results 19, and the erroneous entries are then taken as a basis for a second training of the algorithm 18 in a further step S6. According to step S6, thus, second training is effected, in which also the learning algorithms or learning methods known from the prior art can be used. However, exclusively the erroneous entries from the list of results 19 are taken as a basis for this second training.

In a still further step S7, it can then be examined, which patterns of the overall amount G have not been recognized by the preliminarily trained algorithm 18 according to step S4. These non-detected patterns can then be taken as a basis for a third training according to step S7. As is apparent from Fig. 2, optionally, at least the steps S4 to S7 can be repeated. This can be performed until a predetermined percentage (for example 80% or 90% or 95% or 99%) of the overall amount G of the patterns has been correctly recognized by the algorithm 18.

Claims

1 . Method for training an algorithm (18) for recognizing predetermined patterns in image data, which is provided by means of a camera (3, 4, 5, 6) of a motor vehicle (1 ), including the steps of:

a) providing (S1 ) a test database (17) with test image data for training the algorithm (18), wherein the test image data includes an overall amount (G) of patterns, b) selecting (S2) an exemplary subset (S) of patterns, which are representative of predetermined objects (15), from the overall amount (G) of the patterns, c) first training (S3) of the algorithm (18) based on the selected subset (S) of the patterns and herein providing a preliminarily trained algorithm (18),

d) applying (S4) the preliminarily trained algorithm (18) to the entire test database (17) and herein providing a result list (19) of entries, which respectively specify a pattern recognized by the preliminarily trained algorithm (18) and an object (15) associated with this pattern,

e) discarding (S5) entries from the result list (19), in which the association of the recognized pattern with the object (15) has been correctly performed by the preliminarily trained algorithm (18), and

f) second training (S6) of the preliminarily trained algorithm (18) based on the remaining entries.

2. Method according to claim 1 ,

characterized in that

erroneous entries in the result list (19) are detected, in which the recognized pattern has been associated with an incorrect object (15) by the preliminarily trained algorithm (18), and the second training is performed based on these erroneous entries.

3. Method according to claim 1 or 2,

characterized in that

those entries in the result list (19) are detected, in which a pattern was recognized, which would not have been allowed to be recognized by the preliminarily trained algorithm (18).

4. Method according to any one of claims 1 to 3,

characterized in that

it is examined (S7), which patters of the overall amount (G) have not been recognized by the preliminarily trained algorithm (18), wherein third training of the algorithm (18) is performed based on the non-recognized patterns.

5. Method according to any one of claims 1 to 4,

characterized in that

at least the steps d) to f) are repeated until at least a preset percentage of the overall amount (G) of the patterns has been correctly recognized by the algorithm (18).

6. Camera system (2) for a motor vehicle (1 ), including a camera (3, 4, 5, 6) for

providing images of an environmental region (9, 10, 1 1 , 12) of the motor vehicle (1 ), and including an image processing device (13) with an algorithm (18), which has been trained according to a method according to any one of claims 1 to 5.

7. Motor vehicle (1 ) including a camera system (2) according to claim 6.