CN115482659B - An autonomous decision-making method for agents based on deep reinforcement learning - Google Patents

Abstract

The invention discloses an autonomous decision-making method for intelligent agents based on deep reinforcement learning, which relates to the technical field of autonomous decision-making for intelligent agents. By acquiring basic data for intersections in a banned state, the license plate numbers and right-turn times of all comparison objects are obtained. and vehicle identification; the comparison object is automatically deleted after the T1 time is stored; then all potential objects are initially screened, based on the actual entry object's license plate number, entry time and vehicle identification, and the comparison object's license plate number, right turn time and vehicle Compare the signs to determine all the initial suspects and their corresponding intervals; conduct a secondary review of the initial suspects again, and determine the violating objects through the secondary review, which facilitates the processing of law enforcement departments and solves the problem that cannot be targeted in the existing technology. This issue is addressed and poses a security risk.

Description

An autonomous decision-making method for agents based on deep reinforcement learning

Technical field

The invention belongs to the field of autonomous decision-making technology, and is specifically an autonomous decision-making method for intelligent agents based on deep reinforcement learning.

Background technique

The patent number CN111833597A discloses autonomous decision-making in traffic situations with planned control. A control device for generating steering decisions for autonomous vehicles in traffic scenarios is proposed. The control device includes a first module including a trained self-learning model, the first module being configured to receive data including information about a surrounding environment of the autonomous vehicle, based on the received data by means of the trained self-learning model. to determine the actions to be performed by the autonomous vehicle. The control device includes a second module configured to receive the determined action, receive data including information about an environment surrounding the autonomous vehicle during a limited time range, predict an environmental state for a first time period of the limited time range, A trajectory of the autonomous vehicle is determined based on the received limited time range actions and the environmental state of the first time period, and a signal is sent to control the autonomous vehicle during the first time period based on the determined trajectory.

However, this patent still lacks an important link in autonomous decision-making in traffic scenarios. During driving on existing road conditions, there are often situations where people turn right in advance when going straight at a red light, and then turn right again. Turn to avoid the red light. Of course, under normal circumstances, this type of driving is allowed. However, there are often illegal U-turns after turning right, which causes some safety hazards. In order to solve this problem, we now provide A solution.

Contents of the invention

The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes an intelligent agent autonomous decision-making method based on deep reinforcement learning.

In order to achieve the above object, according to the first embodiment of the present invention, a method for autonomous decision-making of an agent based on deep reinforcement learning is proposed. The method specifically includes the following steps:

Step 1: Obtain basic data for intersections that are in the banned state. The specific method is:

Obtain the pictures of all vehicles turning right when the intersection commanded by the target object is in a prohibited state, and mark them as comparison objects;

Synchronously and automatically obtain the license plate number, right turn time and vehicle identification of all comparison objects; the comparison objects are automatically deleted after storing the T1 time;

Step 2: Then conduct an initial screening of all potential targets. Based on the comparison between the license plate number, entry time and vehicle identification of the actual object and the license plate number, right turn time and vehicle identification of the comparison object, all potential targets are determined. The initial suspect and the corresponding interval;

Step 3: Conduct a second review of the initial suspect. The specific method of the second review is:

S1: Obtain all initial suspects and their corresponding intervals;

S2: Then obtain the road before the turn when the initial suspect turns right into the target object command road. The road before the turn is the road where the initial suspect turned right when entering the target object. Obtain the target lane of the road before the turn. , marking it as crossing the lane;

S3: Obtain the distance from the right turn of the target object's command road to the intersection where the vehicle crosses the lane, and then to the U-turn point of the crossing lane. Mark this distance as the U-turn distance, and the U-turn point refers to where the vehicle can make a U-turn;

S4: Then obtain the distance of the U-turn point from the road before the turn to the target command road, and mark this distance as the within-regulation distance;

S5: Obtain the speed limit value 1 of the crossing lane, divide the U-turn distance by the speed limit value 1 to obtain the U-turn order time;

Then obtain the speed limit value 2 of the road before the turn, divide the distance within the regulation by the speed limit value 2 to get the U-turn time of 2; add the U-turn time to the U-turn time of 2 to get the short-limit time;

S6: Mark the initial suspect whose interval exceeds the short time limit as a speculative object;

S7: Get all speculation objects.

Compared with the prior art, the beneficial effects of the present invention are:

This invention obtains the license plate number, right turn time and vehicle identification of all comparison objects by acquiring basic data for intersections in a banned state; the comparison objects are automatically deleted after storing the T1 time; and then all potential objects are initially screened. , based on the comparison between the license plate number, entry time and vehicle identification of the actual object and the license plate number, right turn time and vehicle identification of the comparison object, determine all initial suspects and their corresponding intervals;

Conduct a secondary review of the initially suspected object again, and determine the illegal object through the secondary review, which facilitates the processing of the law enforcement department and solves the problem that the existing technology cannot deal with the problem and causes potential safety hazards.

Detailed ways

The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

This application provides a method for autonomous decision-making by an agent based on deep reinforcement learning. As the first embodiment of the present invention, the method specifically includes the following steps:

Step 1: Synchronize the status of the subject object. When any driving direction is in the prohibited state, start independent decision-making and remain silent in other states; any driving direction here is the passable direction of the road that the traffic light is responsible for commanding;

The target object here refers to the traffic light, and the status of the target object is the specific real-time status of the traffic light, which is red, green, or yellow; the prohibited status is that the vehicles in the straight direction corresponding to the through lane are in a red light and prohibited from passing;

Step 2: Obtain basic data for intersections that are in the banned state. The specific method is:

Obtain the pictures of all vehicles turning right when the intersection directed by the target object is in the prohibited state, mark them as comparison objects, and automatically obtain the license plate numbers, right turn time and vehicle identification of all comparison objects simultaneously; the right turn time is It corresponds to the specific time point when the comparison object turns right; the comparison object stores the T1 time and is automatically deleted. T1 is a preset value, usually fifteen minutes, but of course it can be set to other values;

The vehicle logo refers to the color of the vehicle hood and the vehicle brand. The vehicle brand can be known through automatic camera recognition of the brand;

Step 3: Then conduct an initial screening of all potential targets. The specific method of initial screening is:

S01: Obtain pictures of all vehicles that enter the road directed by the target object by turning right, mark the corresponding vehicles as actual entry objects through image recognition, and obtain the license plate numbers, entry time and vehicle identification of all actual entry objects; entry time That is the time node when the actual object enters the path directed by the target object;

S02: First, compare any pair of the actual entry object and the comparison object, and compare whether the license plate number and vehicle identification of the actual entry object and the comparison object are consistent. If the comparison is consistent, mark the consistent actual entry object as a preliminary suspect. object;

S03: Synchronize and obtain the interval time based on the entry time of the initial suspect object and the right turn time of the comparison object, where the interval time is determined based on the length of the interval from the entry time to the right turn time;

S04: Obtain all initial suspects and their corresponding intervals;

Step 4: Conduct a second review of the initial suspect. The specific method of the second review is:

S1: Obtain all initial suspects and their corresponding intervals;

S2: Then obtain the road before the turn when the initial suspect turns right into the target object command road. The road before the turn is the road where the initial suspect turned right when entering the target object. Obtain the target lane of the road before the turn. , mark it as a passing lane; the passing lane here satisfies the requirement that the vehicle can turn right into the road directed by the subject object;

S3: Obtain the distance from the right turn of the target object's command road to the intersection where the vehicle crosses the lane, and then to the U-turn point of the crossing lane. Mark this distance as the U-turn distance, and the U-turn point refers to where the vehicle can make a U-turn;

S4: Then obtain the distance of the U-turn point from the road before the turn to the target command road, and mark this distance as the within-regulation distance;

S5: Obtain the speed limit value 1 of the crossing lane, divide the U-turn distance by the speed limit value 1 to obtain the U-turn order time;

Then obtain the speed limit value 2 of the road before the turn, divide the distance within the regulation by the speed limit value 2 to get the U-turn time of 2; add the U-turn time to the U-turn time of 2 to get the short-limit time;

S6: Mark the initial suspect whose interval exceeds the short time limit as a speculative object;

S7: Get all speculation objects;

Step 5: Transmit all speculation objects and their pictures to the smart terminal port of the corresponding regulatory department for easy processing; the speculation objects here refer to the user who did not wait to turn right when going straight on the red light and then turned right again. And it is illegal to make a U-turn without legal position to avoid traffic lights;

As Embodiment 2 of the present invention, based on Embodiment 1, step 4 of this embodiment is slightly different from Embodiment 1, specifically as follows:

Step 4: Conduct a second review of the initial suspect. The specific method of the second review is:

S1: Obtain all initial suspects and their corresponding intervals;

S2: Then obtain the road before the turn when the initial suspect turns right into the target object command road. The road before the turn is the road where the initial suspect turned right when entering the target object. Obtain the target lane of the road before the turn. , mark it as a passing lane; the passing lane here satisfies the requirement that the vehicle can turn right into the road directed by the subject object;

S3: Obtain the distance from the right turn of the target object's command road to the intersection where the vehicle crosses the lane, and then to the U-turn point of the crossing lane. Mark this distance as the U-turn distance, and the U-turn point refers to where the vehicle can make a U-turn;

S4: Then obtain the distance of the U-turn point from the road before the turn to the target command road, and mark this distance as the within-regulation distance;

S5: Obtain the speed limit value 1 of the crossing lane, divide the U-turn distance by the speed limit value 1 to obtain the U-turn order time;

Then get the speed limit value 2 of the road before the turn, divide the distance within the regulation by the speed limit value 2 to get the U-turn time of 2; add the U-turn time to the U-turn time of 2 to get the short time limit; if you cross the lane here When there is no U-turn position, the short time limit is marked as 0;

There is no U-turn position here, which specifically refers to:

Go straight along the passing lane, and there is no U-turn position before and including the second traffic light;

S6: Mark the initial suspect whose interval exceeds the short time limit as a speculative object;

S7: Get all speculation objects;

Of course, as the third embodiment of the present invention, based on the first embodiment, after completing the processing of step five, the following steps need to be performed. The specific method is:

SS1: Obtain all speculation objects and the time points marked as speculation objects, and obtain all speculation objects and speculation time points;

SS2: Obtain all the speculative objects in the approaching stage from the present time, and the number of times they are marked as speculative objects, and remove those whose times are lower than X1. Starting from the time, extrapolate three months forward to get the result;

SS3: Mark the remaining speculative objects as inertia objects, and mark the number of times they are marked as speculative objects as inertia times, and obtain all inertia objects and inertia times;

SS4: Select an inertial object, obtain the period from the first time to the last time it was marked as a speculative object, and the interval between each time it was marked as a speculative object and the last time, and mark it as the inter-investment period value. The corresponding marks are Ji, i=1...n, which means that there are n inter-investment period values, that is, it is marked as a speculation object n+1 times;

SS5: Then automatically calculate the mean value of Ji, mark it as P, and calculate the degree of aggregation D of Ji according to the formula. The specific calculation formula is:

SS6: Define the doubling value based on the degree of polymerization D. When D is less than X2, the doubling value is equal to 2;

When X2≤D≤X3, define the doubling value as 1.5;

When D>X3, define the doubling value as 1;

SS7: Then perform the same processing of steps SS4-SS7 on all inertial objects to obtain the doubled value of all inertial objects;

SS8: Use formulas to calculate the measurement values of all inertial objects. The specific formula is:

Measurement value = times of inertia × doubling value;

SS9: Sort the inertial objects according to their measurement values from large to small, and mark the top 35% corresponding inertial objects as habitual targets;

SS10: When any customary object is marked as a preliminary suspect object in step three of the first embodiment, it will be marked as a semi-speculative object simultaneously, and the semi-speculative object will be transmitted to the smart terminal port of the corresponding regulatory department to facilitate the process. Of course, the semi-speculative object here has no illegal facts as a speculative object, so it can be treated lightly. If it is subsequently marked as a speculative object, it will be punished in the same way as a speculative object.

As Embodiment 4 of the present invention, during specific implementation, Embodiment 1 to Embodiment 3 are integrated and implemented.

Some of the data in the above formula are calculated by removing the dimensions and taking their numerical values. The formula is a formula closest to the real situation obtained through software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are determined by those skilled in the art. It is set according to the actual situation or obtained through a large amount of data simulation.

The above embodiments are only used to illustrate the technical methods of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical methods of the present invention can be modified or equivalently substituted. without departing from the spirit and scope of the technical method of the present invention.

Claims (7)

1. An intelligent agent autonomous decision-making method based on deep reinforcement learning, characterized in that the method specifically includes the following steps: Step 1: Obtain basic data for intersections that are in the banned state. The specific method is: Obtain the pictures of all vehicles turning right when the intersection commanded by the target object is in a prohibited state, and mark them as comparison objects; Synchronously and automatically obtain the license plate number, right turn time and vehicle identification of all comparison objects; the comparison objects are automatically deleted after storing the T1 time; Step 2: Then conduct an initial screening of all potential targets. Based on the comparison between the license plate number, entry time and vehicle identification of the actual object and the license plate number, right turn time and vehicle identification of the comparison object, all potential targets are determined. The initial suspect and the corresponding interval; Step 3: Conduct a second review of the initial suspect. The specific method of the second review is: S1: Obtain all initial suspects and their corresponding intervals; S2: Then obtain the road before the turn when the initial suspect turns right into the target object command road. The road before the turn is the road where the initial suspect turned right when entering the target object. Obtain the target lane of the road before the turn. , marking it as crossing the lane; S3: Obtain the distance from the right turn of the target object's command road to the intersection where the vehicle crosses the lane, and then to the U-turn point of the crossing lane. Mark this distance as the U-turn distance, and the U-turn point refers to where the vehicle can make a U-turn; S4: Then obtain the distance of the U-turn point from the road before the turn to the target command road, and mark this distance as the within-regulation distance; S5: Obtain the speed limit value 1 of the crossing lane, divide the U-turn distance by the speed limit value 1 to obtain the U-turn order time; Then obtain the speed limit value 2 of the road before the turn, divide the distance within the regulation by the speed limit value 2 to get the U-turn time of 2; add the U-turn time to the U-turn time of 2 to get the short-limit time; S6: Mark the initial suspect whose interval exceeds the short time limit as a speculative object; S7: Get all speculation objects. 2. A method for autonomous decision-making of an agent based on deep reinforcement learning according to claim 1, characterized in that the following steps need to be performed before step one, specifically: Synchronize the status of the target object. When the target object is in the banned status, autonomous decision-making is started, and it remains silent in other states. 3. An intelligent agent autonomous decision-making method based on deep reinforcement learning according to claim 2, characterized in that the target object refers to a traffic light; the prohibited sign state means that the vehicle in the straight direction corresponding to the straight lane is at a red light and is prohibited from passing. status. 4. An autonomous decision-making method for an agent based on deep reinforcement learning according to claim 1, characterized in that the right-turn time in step one is the specific time point when the corresponding comparison object turns right; The vehicle logo refers to the vehicle hood color and vehicle brand; and T1 is a preset value. 5. An intelligent agent autonomous decision-making method based on deep reinforcement learning according to claim 1, characterized in that the specific method of initial screening in step two is: S01: Obtain pictures of all vehicles that enter the road directed by the target object by turning right, and mark them as actual entry objects. Obtain the license plate numbers, entry time and vehicle identification of all actual entry objects; the entry time is the actual entry object. The time node when the incoming object enters the road directed by the target object; S02: First, compare any pair of the actual entry object and the comparison object, and compare whether the license plate number and vehicle identification of the actual entry object and the comparison object are consistent. If the comparison is consistent, mark the consistent actual entry object as a preliminary suspect. object; S03: Synchronize and obtain the interval time based on the entry time of the initial suspect object and the right turn time of the comparison object. The interval time is determined based on the length of the interval from the entry time to the right turn time; S04: Obtain all initial suspects and their corresponding intervals. 6. An intelligent agent autonomous decision-making method based on deep reinforcement learning according to claim 1, characterized in that the lane insertion in step S2 satisfies the requirement that the vehicle can turn right into the road directed by the target object; In step S5, if there is no U-turn position after crossing the lane, the short-term time elapsed time is marked as 0. 7. A method for autonomous decision-making of an agent based on deep reinforcement learning according to claim 1, characterized in that after completing the processing of step three, the following steps are required: Transmit all speculation objects to the smart terminal port of the corresponding regulatory department.