FR3093369A1 - LEARNING OF A NETWORK OF NEURONES BY COMPARISONS OF VALUES PRODUCED BY A DRIVING SIMULATOR OF A VEHICLE OPERATING IN PARALLEL - Google Patents

Abstract

A method trains a neural network determining driving decisions for a self-driving virtual vehicle. This method comprises: - a first step (10-20) in which a driving simulator is operated in parallel with a set of environmental data and a route or maneuver to determine first decisions of driving with a set of predefined rules, and the neural network so that it determines second driving decisions, - a second step (30) in which first and second values representative of performance of the first and second driving decisions are estimated conduct, and - a third step (40-80) where the neural network is modified by supervised type learning or reinforcement depending on whether the first value is greater than or less than the second value, then the first ones are repeated (10- 20) to third (40-80) steps as long as a stopping criterion is not satisfied. Figure to be published with abstract: Fig. 2

Description

LEARNING A NEURON NETWORK BY COMPARISON OF VALUES PRODUCED BY A DRIVING SIMULATOR OF A VEHICLE OPERATING IN PARALLEL

Technical field of the invention

The invention relates to land vehicles with at least partially automated (or autonomous) driving and possibly of the automotive type, and more specifically the learning of neural networks which are responsible for determining driving decisions for such vehicles.

State of the art

In what follows, a land vehicle is considered to be driving at least partially automated (or autonomous) when it can be driven on a road in an automated manner (partially or totally (without the intervention of its driver)) during a phase of automated driving, or manually (and therefore with the driver's intervention on the steering wheel and/or the pedals) during a manual driving phase. For example, the automated (or autonomous) driving of a vehicle (with at least partially automated driving) may consist of partially or totally directing this vehicle or providing any type of assistance to a natural person driving this vehicle. This therefore covers any automated (or autonomous) driving, from level 1 to level 5 in the scale of the OICA (International Organization of Automobile Manufacturers).

Furthermore, here the term "land vehicle" means any type of vehicle that can circulate on a road, and in particular a motor vehicle, a utility vehicle, a moped, a motorcycle, a minibus, a coach, a storage robot in a warehouse , or a road vehicle.

Many vehicle driving assistance methods (with at least partially automated driving) aim to adapt the driving of these vehicles according to information determined in their environment from an analysis of data acquired in this last by their sensors. This adaptation is done according to driving decisions which can be determined by a neural network having been previously trained (or trained) for this purpose, and possibly capable of generalization on unlearned cases (even optimization on a given criterion ). For example, when the neural network has been trained (or trained) from trial/error, its learning is said to be “by reinforcement”. Each trial/error is penalized by a bonus/penalty system without the expected solution being explicitly provided to the system (because often not known). Using rules to define the bonus/malus may be an option.

A disadvantage of this type of neural network lies in the fact that it is trained without knowing its minimum level of performance, and therefore the relevance of the driving decisions it takes can vary very significantly from one environment. (corresponding to a particular situation) to another environment (corresponding to another particular situation), or even sometimes may be irrelevant.

The aim of the invention is therefore in particular to improve the situation.

Presentation of the invention

In particular, it proposes for this purpose a learning process intended to train a neural network responsible for determining driving decisions for a virtual vehicle with at least partially autonomous driving.

This learning process is characterized by the fact that it includes:

- a first step in which a driving simulator is operated in parallel, with the same set of environmental data defining an environment of the vehicle and a definition of a route or a maneuver, on the one hand, so that it determines at least a first driving decision for the vehicle with a set of predefined rules, and, on the other hand, the neural network so that it determines at least a second driving decision for the vehicle,

- a second step in which first and second values representative respectively of performance of the first(s) and second(s) driving decisions are estimated, and

- a third step in which the neural network is automatically modified either by supervised type learning when the first value is greater than the second value, or by reinforcement type learning when the first value is less than the second value, then the first to third steps are repeated, each time with a new set of environmental data and a new definition of route or maneuver, as long as a stopping criterion is not satisfied.

Thus, when the stopping criterion is satisfied (except in the event of a collision) it is considered that the neural network has become sufficiently efficient. When the stopping criterion is a collision and it is satisfied, we do not need to go further in the simulation, because we know that the sequence of commands chosen is not the right one.

The learning method according to the invention may comprise other characteristics which may be taken separately or in combination, and in particular:

- the stopping criterion can, for example, be chosen from an overall level of performance, a function of the second values determined successively during the iterations, and exceeding a predefined threshold, an error rate becoming lower than a predefined threshold, an evolution second values, determined successively during the iterations, which tends towards a predefined value and exceeds a predefined threshold, and a number n, of times where the second values determined have been greater than the first corresponding values, which reaches a predefined threshold;

- in its third step, the modification of the neural network by supervised type learning can consist in providing or indicating to the neural network the set of rules that must be respected to correctly carry out a path or a maneuver that has just been carried out ;

- in its third step, the modification of the neural network by reinforcement type learning can consist in modifying a personal representation via an adaptation of an internal structure;

- in its second step, the first and second values can be estimated according to at least one chosen performance criterion;

+ each performance criterion can, for example, be chosen from among vehicle passenger comfort, influence on vehicle travel or maneuvering time, risk incurred by the vehicle, decision stability (as for example the fact that one does not oscillate between a decision to change lanes and a decision to stay in the lane), and the use of maximum accelerations.

The invention also proposes a computer program product comprising a set of instructions which, when it is executed by processing means, is capable of implementing a learning method of the type of that presented above for forming a neural network responsible for determining driving decisions for an at least partially autonomous driving virtual vehicle.

The invention also proposes a learning device intended to train a neural network responsible for determining driving decisions for a virtual vehicle with at least partially autonomous driving.

This learning device is characterized in that it comprises at least one processor and at least one memory arranged to perform the operations consisting of:

- triggering the operation in parallel, with the same set of environment data defining an environment of the vehicle and a definition of a route or a maneuver, on the one hand, of a driving simulator so that it determines at least one first driving decision for the vehicle with a set of predefined rules, and, on the other hand, of the neural network so that it determines at least one second driving decision for the vehicle,

- estimating first and second values representative respectively of performance of the first(s) and second(s) driving decisions,

- modifying the neural network either by supervised type learning when the first value is greater than the second value, or by reinforcement type learning when the first value is less than the second value, then

- repeat these triggering, estimation and modification, each time with a new set of environmental data and a new definition of route or maneuver, as long as a stopping criterion is not satisfied.

The invention also proposes a computer comprising, on the one hand, a driving simulator intended to determine, with a set of predefined rules, driving decisions for a virtual vehicle with at least partially autonomous driving, and, on the other hand , a learning device of the type presented above and coupled to a neural network responsible for determining driving decisions for the vehicle.

Brief description of figures

Other characteristics and advantages of the invention will appear on examination of the detailed description below, and of the appended drawings, in which:

schematically and functionally illustrates an exemplary embodiment of a simulation computer equipped with an exemplary embodiment of a learning device according to the invention and of a neural network to be trained, and

schematically illustrates an example of an algorithm implementing a learning method according to the invention.

Detailed description of the invention

The aim of the invention is in particular to propose a learning method, and an associated learning device DA, intended to allow the learning (or the training or even the training) of a neural network RN responsible for determining driving decisions for a virtual vehicle with at least partially autonomous driving. It will be understood that once the neural network RN has been trained (or trained), it is intended to be implanted in an assistance device intended to assist the driving of a real vehicle of the same type as the virtual vehicle with which it was formed (or trained).

In what follows, it is considered, by way of non-limiting example, that the automated driving (or autonomous) virtual vehicle is of the automotive type. For example, a car. But the invention is not limited to this type of virtual vehicle. It relates in fact to any type of land vehicle with at least partially automated driving and able to circulate on land traffic lanes. Thus, it could also be a utility vehicle, a moped, a motorcycle, a minibus, a coach, a storage robot in a warehouse, or a roads, for example.

It will be noted that the invention is intended to allow the automated driving (at least partially) of vehicles on any portion of road comprising at least one traffic lane delimited by two delimitations.

As mentioned above, the invention proposes in particular a learning method intended to allow the learning (or the training or even the training) of a neural network RN responsible for determining driving decisions for a virtual vehicle at at least partially autonomous driving.

This learning process can be at least partially implemented by the learning device DA which comprises for this purpose at least one processor PR, for example of digital signal (or DSP ("Digital Signal Processor")), and a MD memory.

In the example illustrated without limitation in FIG. 1, the learning device DA is part of a computer OS which also includes a driving simulator SC and the neural network RN to be trained (or trained). But this is not mandatory. Indeed, the learning device DA could comprise its own computer and be coupled to a computer OS comprising a driving simulator SC and the neural network RN to be trained (or trained). Consequently, the learning device DA can be produced in the form of a combination of electrical or electronic circuits or components (or “hardware”) and software modules (or “software”). The memory MD is live in order to store instructions for the implementation by the processor PR of the learning process.

Note that the OS computer can be fixed or portable or can be part of a server. It can be physical or virtual (for example distributed in a cloud (or “cloud”)).

The driving simulator SC is arranged so as to determine, according to a set of predefined rules, a first driving decision for the virtual vehicle each time it is fed with environment data and a definition of a trip or maneuver. These environmental data are similar to those acquired by the sensor(s) fitted to a real vehicle. They are representative of the environment which is located at least in front of this vehicle and, preferably, on at least part of the two lateral sides of this vehicle, as well as possibly behind this vehicle.

Each sensor can comprise at least one camera installed in a front part of the vehicle (for example on the windshield or on the interior mirror) or in a rear part, or at least one ultrasonic sensor or at least one radar or lidar.

As illustrated in FIG. 2, the learning method, according to the invention, comprises three steps carried out in series in a repeated manner.

In a first step 10-20 of the learning process, (the processor PR and the memory MD) feed(s) the driving simulator SC and the neural network RN to be trained (or trained) with the same set of data environment defining an environment of the virtual vehicle (at least in front of the latter) and a definition of a route or a maneuver. This power supply is intended to operate in parallel with this set of environmental data on this defined route or this defined maneuver, on the one hand, the driving simulator SC so that it determines at least a first driving decision d1 for the virtual vehicle (with its set of predefined rules), and, on the other hand, the neural network RN so that it determines at least one second driving decision d2 for the virtual vehicle.

In a second step 30 of the learning method, one (the processor PR and the memory MD) estimates first v1 and second v2 values which are respectively representative of the performance of the first(s) d1 and second(s) d2 driving decisions just determined in the first step 10.

It will be noted that the first v1 and second v2 values can be numeric or alphanumeric.

For example, in this second step 30 one (the processor PR and the memory MD) can estimate the first v1 and second v2 values according to at least one chosen performance criterion.

By way of example, each performance criterion can be chosen from among the comfort of the passengers of the vehicle, the influence on the travel or maneuvering time of the vehicle, the risk incurred by the vehicle, the stability of the decisions (such as for example the fact that we do not oscillate between a decision to change lanes and a decision to stay in the lane), and the use of maximum accelerations. But any other performance criterion known to those skilled in the art can be used here.

The more the comfort of the vehicle passengers is respected, the higher the performance. To do this, you must have good management of acceleration (at least longitudinal and if possible also transverse), braking and the steering angle of the steering wheels of the vehicle. In general, we try to avoid as much as possible longitudinal or lateral oscillations which are sources of discomfort, and excessive accelerations or jolts which are sources of stress or discomfort. For example, it can be checked whether the decided accelerations have remained within a predefined interval of values and/or whether the braking intensities have remained within a predefined interval of values and/or whether the steering angles have remained within a predefined interval of values. .

The shorter the travel or maneuvering time, the higher the performance, provided, of course, that the comfort is not (or only slightly) reduced. To do this, you must have good management of overtaking, speed and acceleration.

The lower the risk involved, the higher the performance, provided, of course, that the vehicle is moving. To do this, you must have good management of the avoidance of obstacles present in the environment of the vehicle and the distances between the vehicle and the obstacles which surround it, as well as a good mastery of the rules of conduct and social.

In a third step 40-80 of the learning process, one (the processor PR and the memory MD) automatically modifies (or corrects) the neural network RN either by supervised type learning when the first value v1 is greater than second value v2, or by reinforcement type learning when first value v1 is less than second value v2.

Then, the first 10-20 to the third 40-80 steps are repeated (the processor PR and the memory MD), each time with a new set of environment data and a new definition of route or maneuver, until a stopping criterion is satisfied.

Thanks to the invention, it is now possible to train (or train) a neural network so that it has a minimum level of performance (higher than that of the SC driving simulator). Indeed, when the stopping criterion is satisfied (except in the event of a collision) it is considered that the neural network has become sufficiently efficient. When the stopping criterion is a collision and it is satisfied, we do not need to go further in the simulation, because we know that the sequence of commands chosen is not the right one and therefore that it is not necessary to continue. Thus, the more the decision system will encounter different situations during its operation in parallel with the SC driving simulator, the more it will be able to adapt to new cases.

For example, the stopping criterion that is used in each third step 40-80 can be chosen from:

- an overall performance level, depending on the second values v2 which have been determined successively during the different iterations, and exceeding a predefined threshold,

- an error rate falling below a predefined threshold,

- an evolution of the second values v2, determined successively during the various iterations, which tends towards a predefined value and exceeds a predefined threshold, and

- a number n, of times where the second values v2 determined have been greater than the first corresponding values v1, which reaches a predefined threshold N.

The overall performance level can, for example, be equal to the sum or the average of the second values v2 determined successively during the different iterations. The predefined threshold is at least equal to the overall performance level of the driving simulator SC. It will be understood that the higher this threshold, the more difficult it will be to reach it. After each iteration, the overall performance level is updated, and when it becomes greater than the predefined threshold, the iterations are stopped and therefore the learning of the RN neural network is stopped. It will be understood that in this case, a minimum number P of iterations must be performed before deciding to stop the process. In fact, at the start of the method, there could be an overall level of performance above the predefined threshold. For example (and not limitingly), the minimum number P can be between 100 and 200.

The error rate can, for example, be equal to the total number of errors divided by the number of iterations. After each iteration, the error rate is updated, and when it becomes lower than the predefined threshold, the iterations are stopped and therefore the learning of the RN neural network is stopped. It will be understood that in this case, a minimum number P of iterations must be performed before deciding to stop the process. Indeed, one could have at the beginning of the process an error rate lower than the predefined threshold. For example (and not limitingly), the minimum number P can be between 100 and 200.

It is by comparing the second values v2 which are successively determined during the different iterations that it is possible to determine whether they gradually (asymptotically) tend towards a predefined value, and it is by comparing each second value v2 with the predefined threshold that it can be determined that during an iteration the second values v2 have exceeded this predefined threshold. The evolution of the second values v2 can be done by calculation, for example by calculating gradients. After each iteration, the evolution is updated and when it tends towards the predefined value and at the same time the last second value v2 becomes greater than the predefined threshold, the iterations are stopped and therefore the learning of the neural network is stopped. RN.

The number n can be determined during each iteration by incrementing a counter (n) by one unit each time a second determined value v2 is greater than the corresponding first value v1. After each incrementation, the current value of counter n is compared to the predefined threshold N, and when this current value of counter n becomes equal to N, the iterations are stopped and therefore the learning of the neural network RN is stopped. For example (and not limitingly), the number N can be between 100 and 200.

It will be noted that in the third step 40-80 the modification of the neural network RN by supervised type learning can, for example, be done by modifying its personal representation via an adaptation of its internal structure (for example by changing weightings between its neurons).

It will also be noted that in the third step 40-80 the modification of the neural network RN by reinforcement-type learning may, for example, consist in providing the latter (RN) with a trend representative of its performance on a route or a maneuver coming from to be performed. For example, the neural network RN can be told whether the route (or maneuver) was good or bad or even average, or the neural network RN can be given a value (for example between 1 and 5 or 1 and 10) representative of the quality of the journey or the maneuver performed, or the neural network RN can be provided with the result of the comparison of each second v2 with the first value v1 corresponding during a trip or manoeuvre.

There will be described below, with reference to FIG. 2, an example of an algorithm implementing the three stages of the learning method described above.

In a first sub-step 10 of the first step 10-20, one (the processor PR and the memory MD) feeds the driving simulator SC and the neural network RN with the same set of environment data defining an environment of the virtual vehicle (at least in front of the latter) and a definition of a route or a maneuver (to be performed), in order to operate them in parallel with this set of environment data on the route (or the manoeuvre) defined. In a second sub-step 20 of the first step 10-20, the driving simulator SC determines at least one first driving decision d1 for the virtual vehicle (with its set of predefined rules), and the neural network RN determines at at least one second driving decision d2 for the virtual vehicle.

In a third sub-step 30 of the second step 30, (the processor PR and the memory MD) estimate(s) first v1 and second v2 values respectively representative of the performance of the first(s) d1 and second(s) d2 driving decisions on the defined route (or manoeuvre).

In a fourth sub-step 40 of the third step 40-80, one (the processor PR and the memory MD) performs a test to compare the first v1 and second v2 values with each other. If the first value v1 is greater than the second value v2 (i.e. v1 > v2), one (the processor PR and the memory MD) automatically modifies the neural network RN by supervised type learning in a fifth sub -stage 50 of the third stage 40-80. On the other hand, if the first value v1 is less than the second value v2, one (the processor PR and the memory MD) automatically modifies the neural network RN by reinforcement-type learning in a sixth sub-step 60 of the third stage 40-80.

Then, in a seventh sub-step 70 of the third step 40-80, one (the processor PR and the memory MD) performs a test to determine if the stopping criterion is satisfied. If not ("no"), the first 10-20 to third 40-80 steps are repeated (the processor PR and the memory MD), each time with a new set of environment data and a new journey (or maneuver) definition, as long as the stopping criterion is not satisfied. On the other hand, in the affirmative (“yes”), one (the processor PR and the memory MD) put an end to the training of the neural network RN in an eighth sub-step 80 of the third step 40-80.

It will also be noted that the invention also proposes a computer program product comprising a set of instructions which, when it is executed by processing means of the electronic circuit (or hardware) type, such as for example the processor PR, is capable of implementing the learning method to train a neural network RN responsible for determining driving decisions for a virtual vehicle with at least partially autonomous driving.

It will also be noted that in FIG. 1 the learning device DA is very schematically illustrated with its random access memory MD and its processor PR which may comprise integrated circuits (or printed circuits), or even several integrated circuits (or printed circuits) connected by wired or wireless connections. By integrated (or printed) circuit is meant any type of device capable of performing at least one electrical or electronic operation. But, as illustrated without limitation in FIG. 1, the learning device DA can also comprise a mass memory MM, in particular for storing the environmental data of the sets that it supplies to the neural network RN to be trained, definitions of journeys or maneuvers, first d1 and second d2 driving decisions of the different iterations, first v1 and second v2 values determined in the different iterations, and any intermediate data involved in all its calculations and processing. Furthermore, this learning device DA can also comprise an input interface IE for receiving at least the first d1 and second d2 driving decisions, to use them in calculations or processing, possibly after having implemented them. form and/or demodulated and/or amplified, in a manner known per se, by means of a digital signal processor PR′. In addition, this learning device DA can also comprise an output interface IS, in particular for the transmission, on the one hand, to the driving simulator SC and to the neural network RN of the sets of environment data and the definitions of route or maneuver, and, on the other hand, to the neural network RN of the learning modifications (or corrections) determined at each iteration.

It will also be noted that one or more steps and/or one or more sub-steps of the driving learning process can be performed by different components. Thus, the driving learning process can be implemented by a plurality of digital signal processors, random access memory, mass memory, input interface, output interface.

Claims (9)

Method for learning a neural network (RN) responsible for determining driving decisions for a virtual vehicle with at least partially autonomous driving, characterized in that it comprises:
- a first step (10-20) in which one operates in parallel, with the same set of environmental data defining an environment of said vehicle and a definition of a route or a maneuver, i) a driving simulator (SC) to determine at least a first driving decision for said vehicle with a set of predefined rules, and ii) said neural network (RN) to determine at least a second driving decision for said vehicle ,
- a second step (30) in which first and second values representative respectively of performance of said first(s) and second(s) driving decisions are estimated, and
- a third step (40-80) in which said neural network (RN) is modified in an automated manner either by supervised type learning when said first value is greater than said second value, or by reinforcement type learning when said first value is less than said second value, then said first (10-20) to third (40-80) steps are repeated, each time with a new set of environmental data and a new definition of route or maneuver, as long as 'a stopping criterion is not satisfied. Method according to Claim 1, characterized in that the said stopping criterion is chosen from among i) an overall level of performance, a function of the said second values determined successively during the said iterations, and exceeding a predefined threshold, ii) an error rate becoming below a predefined threshold, iii) an evolution of said second values, determined successively during said iterations, which tends towards a predefined value and exceeds a predefined threshold, and iv) a number n, of times where said second determined values have been greater than the first corresponding values, which reaches a predefined threshold. Method according to claim 1 or 2, characterized in that in said third step (40-80) said modification of the neural network (RN) by supervised type learning consists in modifying a personal representation via an adaptation of an internal structure. Method according to one of Claims 1 to 3, characterized in that in the said third step (40-80) the said modification of the neural network (RN) by reinforcement-type learning consists in providing the latter (RN) with a trend representative its performance on a journey or a maneuver that has just been carried out. Method according to one of claims 1 to 4, characterized in that in said second step (30) said first and second values are estimated as a function of at least one chosen performance criterion. Method according to Claim 5, characterized in that each performance criterion is chosen from among passenger comfort of said vehicle, influence on travel time or maneuvering time of said vehicle, risk incurred by said vehicle, decision stability and use of maximum accelerations. Computer program product comprising a set of instructions which, when executed by processing means, is capable of implementing the learning method according to one of the preceding claims for forming a neural network (RN ) responsible for determining driving decisions for an at least partially autonomous driving virtual vehicle. Learning device (DA) for training a neural network (RN) responsible for determining driving decisions for a virtual vehicle with at least partially autonomous driving, characterized in that it comprises at least one processor (PR) and at at least one memory (MD) arranged to perform the operations consisting of:
- triggering the operation in parallel, with the same set of environmental data defining an environment of said vehicle and a definition of a route or a maneuver, i) of a driving simulator (SC) so that it determines at least one first driving decision for said vehicle with a set of predefined rules, and ii) said neural network (RN) to determine at least one second driving decision for said vehicle,
- estimating first and second values representative respectively of performance of said first(s) and second(s) driving decisions,
- modifying said neural network (RN) either by supervised type learning when said first value is greater than said second value, or by reinforcement type learning when said first value is less than said second value, then
- repeating said triggering, estimation and modification, each time with a new set of environmental data and a new path or maneuver definition, as long as a stopping criterion is not satisfied. Computer (OS) comprising a driving simulator (SC) intended to determine, with a set of predefined rules, driving decisions for a virtual vehicle with at least partially autonomous driving, characterized in that it further comprises a device for learning device (DA) according to claim 8, coupled to a neural network (RN) responsible for determining driving decisions for said vehicle.