CN116755046B - Multifunctional radar interference decision-making method based on imperfect expert strategy - Google Patents

Abstract

The invention relates to a multifunctional radar interference decision-making method based on imperfect expert strategy, which includes the steps of: obtaining radar status; when judging that the radar status is inconsistent with the radar target status, using an expert intervention discriminant function module to judge whether the radar status belongs to the expert decision-making error status set ; When it is judged that the radar status belongs to the expert decision-making error state set, the main decision-making network in the interference decision-making network is used to select the interference pattern; when the radar status is judged not to belong to the expert decision-making error state set, the expert interference exploration function module is used to judge based on the radar status. Whether the expert strategy participates in the interference decision-making; when it is judged that the expert strategy participates in the interference decision-making, the expert decision-making network module is used to select the interference style; when it is judged that the expert strategy does not participate in the interference decision-making, the main decision-making network in the interference decision-making network is used to select the interference style. This method effectively improves the learning efficiency and decision-making accuracy of the interference decision-making algorithm, and reduces the trial-and-error cost of adversarial games.

Description

A multifunctional radar interference decision-making method based on imperfect expert strategy

Technical field

The invention belongs to the field of radar technology, and specifically relates to a multifunctional radar interference decision-making method based on imperfect expert strategies.

Background technique

With the widespread application of digital technology and artificial intelligence technology in the field of radar, multi-functional radar with multi-tasks and multi-waveforms has become an important means of modern microwave detection. However, for the party being detected, the advantages of radar's high intelligence and flexibility bring more serious threats and challenges to the targets and areas being detected. It is difficult for jammers to take effective jamming measures flexibly and quickly, so they are at a disadvantage. As an important part of radar game confrontation, interference decision-making is the key to whether it can implement effective interference. Therefore, studying adaptive and intelligent interference decision-making methods to deal with radars with cognitive capabilities is of great significance for improving interference effectiveness. At present, intelligent interference decision-making combined with reinforcement learning is one of the research hotspots in the radar field, and is also considered to be one of the effective means to solve the problem of multi-functional radar interference. In recent years, with the continuous development of technology in the field of deep learning, deep reinforcement learning, which combines the advantages of deep learning and reinforcement learning, has performed well in the field of intelligent control decision-making and is widely used in fields such as robot control, autonomous driving, game AI, and natural language processing. Therefore, many scholars have introduced reinforcement learning into the field of radar interference decision-making and proposed various interference decision-making methods based on reinforcement learning.

Li Yunjie and others combined reinforcement learning with the radar countermeasures process for the first time, providing new ideas for radar interference decision-making methods. Zhang Bokai and others proposed an interference decision-making method based on DQN (Deep Q Network) to solve the problem of reduced efficiency of interference decision-making based on Q-learning due to the increase in radar states, which realizes interference decision-making for multiple radar states. Li Huiqin and others used the simulated annealing algorithm and the idea of stochastic gradient descent with hot restart to improve the Q-learning algorithm, improve the exploration and utilization of interference strategies, and achieve faster convergence effects. Zou Weiqi and others introduced A3C (Asynchronous Advantage Actor-Critic, asynchronous advantage action evaluation algorithm) into the field of interference decision-making, improving decision-making time efficiency. Zhu Bakun and others aimed at the problem of the slow convergence speed of the Q-learning algorithm and improved it from the two perspectives of introducing prior knowledge and combining the Dyna architecture to improve the convergence speed of the algorithm. Liu Hongdi and others proposed a two-level interference decision-making algorithm to realize the strategy optimization problem of interference patterns and interference parameters. By analyzing the sample sampling method, Li Yongfeng proposed the DDQN (Double DQN, Double Deep Q Network) interference decision-making method based on supervised sampling, which improved the efficiency and stability of decision-making. Liu Songtao and others further analyzed the DQN algorithm and proposed a radar interference decision-making algorithm based on D3QN (Dueling Double Deep Q Network), which improved the efficiency and accuracy of interference decision-making.

However, as the radar status and jamming action space increase, there is huge pressure on the calculation and storage of the traditional Q-learning algorithm, and the algorithm complexity increases exponentially. This will seriously affect the timeliness of the jammer's decision-making, resulting in The efficiency of jammer decision-making is low; in addition, the existing intelligent jammer decision-making learning method based on reinforcement learning still has the problems of low learning efficiency and high trial-and-error cost in adversarial games.

Contents of the invention

In order to solve the above-mentioned problems existing in the prior art, the present invention provides a multifunctional radar interference decision-making method based on imperfect expert strategy. The technical problems to be solved by the present invention are achieved through the following technical solutions:

The embodiment of the present invention provides a multifunctional radar interference decision-making method based on imperfect expert strategy. The method is based on a multifunctional radar interference decision-making model and utilizes a trained decision-making network to select an interference pattern according to the radar status to interfere with the radar. The radar generates a new radar state; the trained decision-making network includes an expert decision-making network module, an interference decision-making network, an expert intervention discriminant function module and an expert interference exploration function module; the method includes the steps:

Get radar status;

When it is determined that the radar state is inconsistent with the radar target state, the expert intervention discriminant function module is used to determine whether the radar state belongs to the expert decision-making error state set;

When it is judged that the radar state belongs to the expert decision-making error state set, then the main decision-making network in the interference decision-making network is used to select the interference pattern; when it is judged that the radar state does not belong to the expert decision-making error state set, then according to the radar state, the interference pattern is selected The expert interference exploration function module determines whether the expert strategy participates in interference decision-making;

When it is judged that the expert strategy participates in the interference decision-making, the expert decision-making network module is used to select the interference pattern; when it is judged that the expert strategy does not participate in the interference decision-making, the main decision-making network in the interference decision-making network is used to select the interference pattern.

In one embodiment of the present invention, the multifunctional radar interference decision model is:

<S,J,P,R,T>;

Among them, S is the radar state space, S={s ₁ , s ₂ ,..., s _N }, s _n , n=1, 2,..., N is the radar state in S, and N is the number of radar states. ; J is the interference pattern space, J={j ₁ ,j ₂ ,…,j _M }, j _m ,m=1,2,…, M is the interference pattern of the jammer in J, M is the interference pattern individual number; P:S×J×S→[0,1] is the state transition probability, T is the end signal, and R is the reward function;

The reward function R is defined as:

Among them, r _t is the interference gain at time t, t is the time before interference is implemented, t+1 is the time when radar signals are detected again after interference is implemented, is the radar threat level before jamming,/> is the radar threat level after interference, s _target is the radar target state, s _t is the state of the radar when interference is implemented, and s _t+1 is the state of the radar after interference is implemented.

In one embodiment of the present invention, the method for constructing the expert decision-making network module includes the steps:

Expert data is used as a training sample set; wherein, through the behavioral cloning method, the current multi-functional radar state in the expert data is used as the input of the initial expert decision-making network, and the probability distribution of the interference pattern adopted in the current multi-functional radar state is used as the initial The output of the expert decision-making network is labeled with the interference pattern corresponding to the multi-function radar state in the expert data;

Construct an initial expert decision-making network represented by a deep neural network;

The initial expert decision-making network is trained using the training sample set, and the cross-entropy loss function is back-propagated to update the parameters of the initial expert decision-making network to obtain the expert decision-making network module.

In one embodiment of the present invention, the expert data is expressed in the form of a trajectory:

Γ={(s _i ,j _i )|i=1,2,...,N _e };

Among them, s _i is the multifunctional radar state, j _i is the jamming pattern of the jammer, j _i ∈J, J is the jamming pattern space, (s _i , j _i ) is the i-th expert knowledge, and N _e represents the individual expert knowledge. number;

The cross-entropy loss function is:

Among them, M is the number of interference patterns, j _k is the kth interference pattern, is the probability distribution of the interference pattern adopted in the current multifunctional radar state, and p _e (·|s) is the probability distribution of the interference pattern corresponding to the multifunctional radar state in the expert data.

In one embodiment of the present invention, the interference decision-making network includes a main decision-making network and a target decision-making network. The main decision-making network and the target decision-making network have the same structure, and both include: a third fully connected layer, a second activation layer layer, the fourth fully connected layer, the third activation layer, the fifth fully connected layer, the sixth fully connected layer and the addition module, where,

The third fully connected layer, the second activation layer, the fourth fully connected layer, and the third activation layer are connected in sequence;

The input terminal of the fifth fully connected layer and the input terminal of the sixth fully connected layer are both connected to the output terminal of the third fully connected layer;

The output terminal of the fifth fully connected layer and the output terminal of the sixth fully connected layer are both connected to the input terminal of the adding module;

The output terminal of the adding module serves as the output terminal of the main decision-making network or the target decision-making network.

In one embodiment of the present invention, the trained decision-making network is obtained by training a decision-making network. The training method of the decision-making network includes the steps:

S2041. Set the total number of training games, the maximum number of games in a single round and the exploration factor, and initialize the network parameters, learning rate, experience pool and sampling batch size of the interference decision-making network;

S2042. Obtain the current radar state. When it is determined that the current radar state is inconsistent with the radar target state, input the current radar state into the expert intervention discriminant function module; when it is determined that the current radar state is consistent with the radar target state , start a new round of training until the number of training times reaches the total number of training games;

S2043. When the expert intervention discriminant function module determines that the current radar state belongs to the expert decision error state set, the main decision network in the interference decision network is used to select the current interference pattern to implement interference; when the expert intervention discriminant function module determines that the If the current radar state does not belong to the expert decision-making error state set, then according to the current radar state, the expert interference exploration function module is used to determine the participation degree of the expert strategy in interference decision-making; when it is judged that the expert strategy participates in the interference decision-making, the expert is used The decision-making network module selects the current interference pattern to implement interference; when it is judged that the expert strategy does not participate in interference decision-making, the main decision-making network in the interference decision-making network is used to select the current interference pattern to implement interference;

S2044. Obtain the post-interference radar state, and evaluate the current interference benefit in combination with the current radar state. Use the current radar state, the current interference pattern, the current interference benefit, the post-interference radar state and the current end signal as The combination is stored in the experience pool;

S2045. Adopt the sampling method of priority experience replay, sample experience samples from the experience pool according to the sampling batch size to train and update the main decision-making network, and use the target loss function to back propagate to update the main decision-making network parameters;

S2046. Update the target decision network parameters by combining the hyperparameters and the main decision network parameters;

S2047. Store expert strategy error information into the expert decision-making error state set;

S2048. When it is judged that the number of games in a single round is less than the maximum number of games in a single round, return to step S2042; when it is judged that the number of games in a single round is greater than or equal to the maximum number of games in a single round, return to step S2041 until the number of training times reaches the total number of training games, and the trained decision network.

In one embodiment of the present invention, the target loss function is:

Among them, D3QN is the D3QN algorithm, θ is the main decision-making network parameter, Q (s _t ,j _t ; θ) is the interference value of j _t output by the main decision-making network under s _t , s _t is the current radar state, j _t is the current interference pattern, r _t is the current interference profit, Q _T is the target decision-making network, s _t+1 is the radar state after interference, J is the interference pattern space, j is the interference pattern of the jammer, Q(s _t+1 , j; θ) is the interference value output by the main decision-making network after interference, taking j under s _t+1 , θ ^- is the target decision-making network parameter;

The update formula of the main decision network parameters is:

Among them, θ _new is the main decision-making network parameter after the update, θ _old is the main decision-making network parameter before the update, l is the learning rate, is the gradient of the target loss function with respect to θ _old ;

The update formula of the target decision network parameters is:

in, Determine network parameters for the updated target,/> are the target decision network parameters before updating, and τ is the hyperparameter.

In one embodiment of the present invention, the output strategy of the trained decision-making network is:

in, For the expert intervention discriminant function module,/> is the expert interference exploration function module, π _e is the expert strategy in the expert decision-making network module, π _q is the interference strategy in the interference decision-making network, and s is the radar state.

In one embodiment of the present invention, the expert intervention discriminant function module is defined as:

Among them, s is the radar state, is the expert decision-making error state set,/> Indicates the use of expert strategy π _e to make decisions,/> Indicates the use of interference strategy π _q to make decisions.

In one embodiment of the present invention, the expert interference exploration function module is defined as:

Among them, s is the radar state, is the expert decision-making error state set, ξ is a random number from 0 to 1, ε _e is the exploration factor, Indicates that expert strategy π _e participates in decision-making,/> Indicates the use of interference strategy π _q to explore interference patterns in radar state s.

Compared with the existing technology, the beneficial effects of the present invention are:

1. The present invention proposes a multifunctional radar interference decision-making method with imperfect expert strategy. Based on the traditional deep reinforcement learning interference decision-making algorithm, the expert knowledge accumulated in the field of radar countermeasures is integrated into the interference decision-making method, and combined with the expert decision-making network module and The interference decision-making network jointly performs interference decisions, effectively improves the learning efficiency and decision-making accuracy of the interference decision-making algorithm, and reduces the trial and error cost of adversarial games.

2. The multifunctional radar interference decision-making method of the present invention takes into account the harshness of the optimal conditions of the expert strategy. On the premise of ensuring that the expert strategy provides security and exploration guidance for the decision-making network, the expert knowledge discrimination processing module and the expert interference exploration function are introduced. The module uses the decision-making network to learn and modify expert knowledge of different qualities, which improves the accuracy of decision-making and improves the timeliness of decision-making.

Description of the drawings

Figure 1 is a schematic flowchart of a multifunctional radar interference decision-making method based on imperfect expert strategies provided by an embodiment of the present invention;

Figure 2 is a schematic diagram of a method for obtaining a trained decision-making network provided by an embodiment of the present invention;

Figure 3 is a schematic structural diagram of an initial expert decision-making network provided by an embodiment of the present invention;

Figure 4 is a schematic structural diagram of an interference decision-making network provided by an embodiment of the present invention;

Figure 5 is a schematic diagram of the training implementation framework of the decision-making network provided by the embodiment of the present invention;

Figure 6 is a radar state transition relationship diagram provided by an embodiment of the present invention;

Figure 7 is a decision accuracy curve chart provided by an embodiment of the present invention;

Figure 8 is another decision-making accuracy curve provided by an embodiment of the present invention.

Detailed ways

The present invention will be described in further detail below with reference to specific examples, but the implementation of the present invention is not limited thereto.

Embodiment 1

Please refer to Figure 1. Figure 1 is a schematic flow chart of a multifunctional radar interference decision-making method based on imperfect expert strategies provided by an embodiment of the present invention. This method is based on the constructed multifunctional radar interference decision-making model, and uses the trained decision-making network to select the interference pattern according to the radar state to interfere with the radar, so that the radar generates a new radar state. The trained decision-making network includes expert decision-making network module, interference decision-making network, expert intervention discriminant function module and expert interference exploration function module. The method includes steps:

S101. Obtain radar status.

S102. When it is determined that the radar state is inconsistent with the radar target state, use the expert intervention discriminant function module to determine whether the radar state belongs to the expert decision-making error state set. When the radar status is judged to be consistent with the radar target status, no interference decision is required at this time.

S103. When it is judged that the radar state belongs to the expert decision-making error state set, use the main decision-making network in the interference decision-making network to select the interference pattern; when it is judged that the radar state does not belong to the expert decision-making error state set, use the expert interference exploration function module according to the radar state Determine the degree of participation of expert strategies in interfering with decision-making, that is, determine whether expert strategies participate in decision-making.

S104. When it is judged that the expert strategy participates in the interference decision-making, the expert decision-making network module is used to select the interference pattern; when it is judged that the expert strategy does not participate in the interference decision-making, the main decision-making network in the interference decision-making network is used to select the interference pattern.

Please refer to Figure 2. Figure 2 is a schematic diagram of a method for obtaining a trained decision-making network provided by an embodiment of the present invention. The method includes steps:

S201. Construct a multifunctional radar interference decision-making model.

Specifically, in a complex electromagnetic environment, the game process between non-cooperative radar countermeasures is uncertain and dynamic. The radar state sequence satisfies Markov property, that is, the countermeasures system only relies on the current radar status and the interference strategy of the jammer. . Therefore, the multifunctional radar interference decision-making problem is constructed as Markov Decision Process (Markov DecisionProcess, MDPs). MDPs is a framework for formal sequence decision-making problems for stochastic dynamic systems with Markov properties. It is described by a five-tuple <S, J, P, R, T>, where S is the state set, It can be used to represent the radar state space. J is the action set, which can be used to represent the interference pattern space. P:S×J×S→[0,1] is the state transition probability, that is, after the agent takes action j _t in the current state s _t , the probability p(s _{t +1} |s _t ,a _t ) of the environment state transitioning to s t+1, R is the reward function, and T is the end signal. Under the framework of MDPs, its core goal is to find the optimal strategy, that is, the best mapping π:S→A from state space to action space, to obtain the maximum expected reward:

Among them, π ^* represents the optimal strategy, π represents the strategy, γ is the discount factor, t represents the time step, T _end represents the step size at the end of a round of interaction, and π _t represents the strategy adopted at time t.

In the multifunctional radar interference decision-making model, the radar state space is defined as S = {s ₁ , s ₂ ,..., s _N }, which represents the radar threat information, that is, the radar state; the radar target state is recorded as s _target , indicating that the interference purpose has been achieved; N represents the number of radar states. The interference pattern space is defined as J={j ₁ ,j ₂ ,…,j _M }, which represents the set of interference patterns that the jammer can adopt, such as noise suppression interference, dense false target interference, smart noise interference, comb spectrum interference, etc. Among them, M represents the number of interference patterns. The reward function R represents the interference effectiveness evaluation after specific interference is adopted. R is defined based on the change in radar threat level before and after interference:

(1) The threat level is reduced to the minimum, that is, the target radar state s _target is reached, and the reward is set to r=100;

(2) The threat level is reduced, but not to the minimum, and the reward is set to r=-1;

(3) The threat level remains unchanged or increases, and the reward is set to r=1.

Based on the above definition, the multifunctional radar interference decision model based on Markov decision process is:

<S,J,P,R,T>;

Among them, S is the radar state space, S={s ₁ , s ₂ ,..., s _N }, s _n , n=1, 2,..., N is the radar state in S, and N is the number of radar states. ; J is the interference pattern space, J={j ₁ ,j ₂ ,…,j _M }, j _m ,m=1,2,…, M is the interference pattern of the jammer in J, M is the interference pattern individual number; P:S×J×S→[0,1] is the state transition probability, T is the end signal, and R is the reward function; the reward function R is defined as:

Among them, r _t is the interference gain at time t, t is the time before interference is implemented, t+1 is the time when radar signals are detected again after interference is implemented, is the radar threat level before jamming,/> is the radar threat level after interference, s _target is the radar target state, s _t is the state of the radar when interference is implemented, and s _t+1 is the state of the radar after interference is implemented.

Based on the above-mentioned multifunctional radar jamming decision-making model, the entire radar countermeasures process can be described as follows: the jammer determines the radar status information by detecting signals, then adopts a certain jamming style according to the jamming strategy, and analyzes the changes in the radar threat level to obtain rewards. After being jammed, The radar takes anti-interference measures and the radar state is transferred to a new state. During the continuous confrontation process, the jammer continuously updates the jamming strategy through reward benefits to obtain the maximum jamming benefit and achieve adaptive jamming of multi-functional radars.

S202. Parameterize the expert knowledge information into an expert decision-making network model.

The expert knowledge information in the field of radar countermeasures is parameterized through behavioral cloning, and the parameterized expert knowledge information in the field of radar countermeasures is transformed into an expert decision-making network model represented by a deep neural network. Specific steps include:

S2021. Use expert data as a training sample set; among which, through the behavioral cloning method, the current multi-function radar state in the expert data is used as the input of the initial expert decision-making network, and the probability distribution of the interference pattern adopted in the current multi-function radar state is used as the initial The output of the expert decision-making network is labeled with the interference pattern corresponding to the multifunctional radar state in the expert data.

Specifically, behavioral cloning is one of the three main imitation learning methods. Its learning task is to imitate problem-solving strategies from expert data, that is, expert strategies, and its essence is supervised learning. Assuming that expert knowledge information (ie expert data) in the field of radar countermeasures is Γ, the expert data is expressed in the form of a trajectory as:

Γ={(s _i ,j _i )|i=1,2,...,N _e };

Among them, s _i is the multifunctional radar state, j _i is the interference pattern of the jammer, j _i ∈J, J is the interference pattern space, (s _i , j _i ) is the i-th expert knowledge, that is, under the state s _i Take interference action j _i , N _e represents the number of expert knowledge.

Behavioral cloning uses expert data Γ as a training sample set, the current multifunctional radar state s in expert data Γ as the input of the initial expert decision-making network model, and the probability distribution of the interference pattern adopted in this state s is the output, and the interference pattern corresponding to the multifunctional radar state in the expert data Γ is used as the label, where the label is in the form of One-hot encoding, which can also be recorded as a probability distribution p _e (·|s).

S2022. Construct an initial expert decision-making network represented by a deep neural network.

The initial expert decision-making network is represented by a deep neural network, including a fully connected layer, an activation layer, and a normalization layer. The number and hierarchical structure of the fully connected layer, activation layer, and normalization layer can be set according to the actual situation.

Please refer to Figure 3. Figure 3 is a schematic structural diagram of an initial expert decision-making network provided by an embodiment of the present invention. The initial expert decision-making network in Figure 3 includes a first fully connected layer, a first activation layer, and a second fully connected layer connected in sequence. layer and normalization layer. Specifically, the first activation layer includes a ReLU activation layer, and the normalization layer includes a softmax layer.

S2023. Use the training sample set to train the initial expert decision-making network, and back-propagate the cross-entropy loss function to update the parameters of the initial expert decision-making network to obtain the expert decision-making network module. Specific training steps include:

2023a) Randomly select expert data samples and send them to the initial expert decision-making network to obtain the probability distribution of the interference pattern adopted by the multifunctional radar in state s

2023b) Calculate the cross-entropy loss function:

Among them, M is the number of interference patterns, j _k is the kth interference pattern, is the probability distribution of the interference pattern adopted in the current multifunctional radar state, and p _e (·|s) is the probability distribution of the interference pattern corresponding to the multifunctional radar state in the expert data.

2023c) Back-propagate the cross-entropy loss function Loss to update the parameters of the initial expert decision-making network. If the loss function converges, the expert decision-making network module is obtained.

S203. Construct an interference decision-making network based on D3QN.

The interference decision-making network is constructed based on the D3QN algorithm, including the main decision-making network Q(s,j;θ) and the target decision-making network _QT (s,j; ^θ- ). The main decision-making network Q(s,j;θ) is used for interference decision-making. , the target decision network Q _T (s, j; θ ^- ) is used to update the main decision network parameters. The main decision-making network Q(s,j;θ) and the target decision-making network _QT (s,j;θ- ⁾ have the same network structure, both including fully connected layers and activation layers, the number and hierarchical structure of fully connected layers and activation layers It can be set according to the actual situation.

Please refer to Figure 4, which is a schematic structural diagram of an interference decision-making network provided by an embodiment of the present invention. As shown in Figure 4, both the main decision-making network and the target decision-making network include: the third fully connected layer, the second activation layer, the fourth fully connected layer, the third activation layer, the fifth fully connected layer, the sixth fully connected layer and Addition module. Among them, the third fully connected layer, the second activation layer, the fourth fully connected layer, and the third activation layer are connected in sequence; the input end of the fifth fully connected layer and the input end of the sixth fully connected layer are both connected to the third fully connected layer The output terminal; the output terminal of the fifth fully connected layer and the output terminal of the sixth fully connected layer are both connected to the input terminal of the addition module, and the output of the fifth fully connected layer and the output of the sixth fully connected layer are performed using addition operations. Addition; the output terminal of the addition module serves as the output terminal of the main decision-making network or the target decision-making network. Specifically, the second activation layer and the third activation layer both include ReLU activation layers; the fifth fully connected layer has 1 node, and the sixth fully connected layer has M nodes, where M represents the number of interference patterns.

S204. Construct a decision-making network and train the decision-making network to obtain a trained decision-making network.

The decision-making network constructed in this embodiment includes: (1) the expert decision-making network model constructed in step S202, (2) the interference decision-making network constructed in step S203, (3) expert intervention discriminant function module (4) Expert interference exploration function module Among them, the expert decision-making network module represents the expert strategy π _e ; the interference decision-making network represents the interference strategy π _q of online learning; the expert intervention discriminant function module/> Based on the radar status s detected by the current jammer, it is decided whether the expert decision-making network module decides on the jamming action; the expert jamming exploration function module/> The participation degree of the expert strategy is controlled based on the current radar status s detected by the jammer.

In this embodiment, the expert decision-making network module and the interference decision-making network jointly make decisions to form a hybrid interference decision-making mechanism, which has two purposes: first, to re-explore and learn erroneous expert knowledge to improve the efficiency of interference decision-making; second, to Participate in expert decision-making, explore possibilities other than expert strategies based on probability, and avoid expert strategies as suboptimal solutions. The final output strategy π of the decision-making network (that is, the output strategy of the trained decision-making network) is expressed as:

Among them, π _e is the expert strategy in the expert decision-making network module, π _q is the interference strategy in the interference decision-making network, and s is the radar state; For the expert intervention discriminant function module,/> The knowledge base established during the game confrontation between the two parties is used to record the error information of the expert strategy, and the expert decision-making error state set is expressed as/> Then the expert intervention discriminant function module is defined as/> Represents the use of expert strategy π _e to make decisions, on the contrary Indicates the use of interference strategy π _q to make decisions;/> Exploring function modules for expert interference, /> Defined as/> Where ξ represents a random number from 0 to 1, ε _e represents the exploration factor, Indicates that expert strategy π _e participates in decision-making, on the contrary/> Indicates the use of interference strategy π _q to explore interference patterns in state s.

It should be noted that in the hybrid interference decision-making mechanism, the interference decision-making network is trained using the data collected interactively by the final policy π, and part of it is generated by the expert policy π _e in some states. In these states There are strategic differences between the expert strategy π _e and the interference strategy π _q This strategy difference affects the state distribution D _π (s) and/> under the mixed strategy and interference strategy When the difference is too large, it will directly lead to the instability of decision-making network training and the decline of decision-making performance. The difference in state distribution between the mixed strategy and the interference strategy is constrained by the difference in their strategies. Then the difference in state distribution between the final output strategy and the interference strategy can be obtained as follows:

in, is the expectation, s～π means that s obeys the strategy π distribution, γ is the discount factor, called intervening factors.

It can be seen from the above formula that the upper limit of the state distribution difference is constrained by the strategy difference and intervention factors. The strategy difference is uncontrollable, but the state distribution difference can be reduced by reducing the participation of the expert strategy. Therefore, in this embodiment, the expert strategy The proportion of participation in decision-making is gradually reduced, which improves the stability of decision-making network training, which is specifically reflected in the gradual reduction of the exploration factor ε _e .

Please refer to Figure 5. Figure 5 is a schematic diagram of the training implementation framework of the decision-making network provided by an embodiment of the present invention. The training method of the decision-making network includes the steps:

S2041. Set the total number of training games N _round , the maximum number of games in a single round N _max and the exploration factor ε _e , and initialize the network parameters of the interference decision-making network, the learning rate l, the experience pool H and the sampling batch size N _b .

S2042. Obtain the current radar state s _t . When it is determined that the current radar state s _t is inconsistent with the radar target state s _target , that is, s _t ≠s _target , then input the current radar state into the expert intervention discriminant function module and go to step S2043; when it is determined The current radar state s _t is consistent with the radar target state s _target , and a new round of training starts until the number of training times reaches the total number of training games N _round .

S2043, if Then the current interference pattern j _t is selected through the interference decision-making network and interference is implemented; otherwise, the interference exploration function controls the expert strategy to participate in decision-making.

Specifically, when the expert intervention discriminant function module determines that the current radar state s _t belongs to the expert decision-making error state set Then use the main decision-making network in the interference decision-making network to select the current interference pattern j _t to implement interference; when the expert intervention discriminant function module determines that the current radar state s _t does not belong to the expert decision-making error state set/> Then according to the current radar state s _t , the expert interference exploration function module is used to judge the participation degree of the expert strategy in interference decision-making; when it is judged that the expert strategy is involved in the interference decision-making, the expert decision-making network module is used to select the current interference style j _t to implement interference; when it is judged If the expert strategy does not participate in interference decision-making, the main decision-making network in the interference decision-making network is used to select the current interference pattern j _t to implement interference.

S2044. Obtain the post-interference radar state s _t+1 and evaluate the current interference gain r _t based on the current radar state. Combine the current radar state s _t , current interference pattern j _t , current interference gain r _t , and post-interference radar state s _{t+ 1} and the current end signal done are stored in the experience pool H as a combination, that is, <s _t ,j _t ,r _t ,s _t+1 ,done>.

S2045. Adopt a sampling method that prioritizes experience playback, and sample experience samples from the experience pool H according to the sampling batch size. The number of experience samples is the same as the sampling batch size N _b to train and update the main decision-making network Q (s, j; θ) , and use the target loss function to back propagate to update the main decision network parameters θ. Among them, the calculation formula of the target loss function is:

Among them, D3QN is the D3QN algorithm, θ is the main decision-making network parameter, Q (s _t ,j _t ; θ) is the interference value of j _t output by the main decision-making network under s _t , s _t is the current radar state, j _t is the current interference pattern, r _t is the current interference profit, Q _T is the target decision-making network, s _t+1 is the radar state after interference, J is the interference pattern space, j is the interference pattern of the jammer, Q(s _t+1 , j; θ) is the interference value of j taken under s _t+1 output by the main decision-making network, and θ ^- is the target decision-making network parameter.

The update formula of the main decision network parameters is:

Among them, θ _new is the main decision-making network parameter after the update, θ _old is the main decision-making network parameter before the update, l is the learning rate, is the gradient of the target loss function with respect to θ _old .

S2046. Update the target decision network parameters by combining the hyperparameters and the main decision network parameters.

For example, set the hyperparameter τ, use the hyperparameter τ as a weight and perform a weighted average with the main decision network parameter θ to update the target decision network parameter θ ^- . The update formula is as follows:

in, Determine network parameters for the updated target,/> are the target decision network parameters before updating, and τ is the hyperparameter.

S2047. Record the expert strategy error information and store the expert strategy error information into the expert decision-making error status set.

S2048. When it is judged that the number of games in a single round is less than the maximum number of games in a single round N _max , return to step S2042 ; when it is judged that the number of games in a single round is greater than the maximum number of games in a single round , return to step S2041 until the number of training times reaches the total number of training games N _round , and training is obtained Good decision-making network. Among them, when returning to step S2041 to start a new round of training, the exploration factor ε _e is smaller than the exploration factor ε _e in the previous round of training, that is, the exploration factor ε _e gradually decreases as the number of training times increases.

Further, the trained decision-making network is used to make multi-functional radar interference decisions according to the method shown in Figure 1.

This embodiment proposes a multifunctional radar interference decision-making method based on imperfect expert strategies. Based on the traditional deep reinforcement learning interference decision-making algorithm, the expert knowledge accumulated in the field of radar countermeasures is integrated into the interference decision-making method, combining the expert decision-making network module and interference decision-making method. The decision-making network jointly performs interference decisions, effectively improving the learning efficiency and decision-making accuracy of the interference decision-making algorithm, and reducing the trial-and-error cost of adversarial games.

The multifunctional radar interference decision-making method of this embodiment takes into account the harshness of the optimal conditions of the expert strategy. On the premise of ensuring that the expert strategy provides security and exploration guidance for the decision-making network, an expert knowledge discrimination processing module and an expert interference exploration function module are introduced. , using the decision-making network for learning and correction of expert knowledge of different qualities, which improves the accuracy of decision-making and improves the timeliness of decision-making.

The effect of this embodiment can be further explained through the following simulation experiments.

In order to verify the performance of the method proposed in this embodiment under complex conditions, it is assumed that the jammer can emit 10 different interference patterns J={j ₁ , j ₂ ,..., j ₁₀ }, and the multifunctional radar state space contains 16 states S = {s ₁ , s ₂ ,…, s ₁₆ }, corresponding to 16 threat levels, of which s ₁ has the highest threat level, s ₁₆ is the target radar state s _target , and has the lowest threat level. Please refer to Figure 6 for the conversion relationship of each radar state. Figure 6 is a radar state transition relationship diagram provided by an embodiment of the present invention. The jamming task of the jammer is to transfer any state of the multi-function radar to the radar target state s _target . The reward function is expressed as:

in, and/> Indicates the radar threat level before and after interference.

In order to compare the performance improvement of the method proposed in this embodiment (abbreviated as ED3QN_PER) relative to the current interference decision-making algorithm based on deep reinforcement learning, from the two dimensions of the number of single games and decision accuracy, it was compared with deep Q learning (DQN) and priority based on deep Q learning (DQN). The decision-making algorithms of deep Q learning with experience replay (DQN_PER), deep double Q learning with priority experience replay (DDQN_PER), and deep duel double Q learning with priority experience replay (D3QN_PER) are compared and analyzed. The decision results are shown in Figure 7. A decision-making accuracy curve provided by an embodiment of the present invention. From the perspective of the decision accuracy curve, the method proposed in this embodiment introduces expert strategies to guide the interference decision-making. During the entire training period, its interference decision-making accuracy is basically better than other algorithms, which is beneficial to the security of the game process. improved.

In order to analyze the performance of the method proposed in this embodiment under expert strategies of different qualities, the four levels of expert strategies of high, medium, low, and poor are used as a reference to analyze the impact of expert strategies of different qualities on algorithm performance. Please refer to the decision-making results. Refer to Figure 8, which is another decision-making accuracy curve provided by an embodiment of the present invention. Judging from the decision-making accuracy curve, under the influence of expert strategies of different qualities, the convergence speed and accuracy of decision-making are excellent, and as the quality of expert strategies improves, the convergence speed and accuracy of decision-making also increase.

In summary, this embodiment aims at the problems of long training cycle and high trial and error cost of the existing interference decision-making algorithm based on reinforcement learning. It introduces expert knowledge to improve the learning efficiency of the interference decision-making algorithm, reduces the risk of trial and error, and targets In the non-cooperative game confrontation between radar and jammers in a complex electromagnetic environment, it is difficult to obtain the optimal expert jamming strategy. Under the conditions of imperfect expert knowledge-assisted decision-making, the efficiency and accuracy of jamming decision-making are improved.

The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be concluded that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of them should be regarded as belonging to the protection scope of the present invention.

Claims (10)

1. A multifunctional radar interference decision method of imperfect expert strategy is characterized in that the method is based on a multifunctional radar interference decision model, and a trained decision network is utilized to select an interference pattern according to radar states so as to implement interference on the radar, so that the radar generates new radar states; the trained decision network comprises an expert decision network module, an interference decision network, an expert interference discrimination function module and an expert interference exploration function module; the method comprises the steps of:

acquiring a radar state;

when the radar state is inconsistent with the radar target state, judging whether the radar state belongs to an expert decision error state set or not by utilizing the expert intervention discriminant function module;

when the radar state is judged to belong to an expert decision error state set, selecting an interference pattern by using a main decision network in the interference decision network; judging whether an expert strategy participates in interference decision or not by utilizing the expert interference exploration function module according to the radar state when the radar state is judged not to belong to the expert decision error state set;

when judging that the expert strategy participates in the interference decision, selecting an interference pattern by using the expert decision network module; and when judging that the expert strategy does not participate in the interference decision, selecting an interference pattern by using a main decision network in the interference decision network.

2. The imperfect expert policy multi-functional radar interference decision method according to claim 1, wherein the multi-functional radar interference decision model is:

<S,J,P,R,T>；

wherein S is a radar state space, s= { S ₁ ,s ₂ ,…,s _N }，s _n N=1, 2,..n is the radar status in S, N is the number of radar states; j is the interference pattern space, j= { J ₁ ,j ₂ ,…,j _M }，j _m M=1, 2,., M is the interference pattern of the interfering machine in J, M is the number of interference patterns; p is SxJxS → [0,1 ]]The state transition probability is represented by T, the ending signal is represented by T, and the reward function is represented by R;

the bonus function R is defined as:

wherein r is _t For the interference gain at time t, t is the time before interference is implemented, t+1 is the time after interference is implemented and radar signals are again detected,for pre-interference radar threat level, +.>For post-interference radar threat level s _target Is radar target state s _t To implement the radar state at interference s _t+1 To implement the radar state after the disturbance.

3. The method for multi-functional radar interference decision making with imperfect expert strategy according to claim 1, wherein the method for constructing the expert decision network module comprises the steps of:

taking expert data as a training sample set; the method comprises the steps of using a current multifunctional radar state in expert data as input of an initial expert decision network, using probability distribution of interference patterns adopted in the current multifunctional radar state as output of the initial expert decision network, and using interference patterns corresponding to the multifunctional radar state in the expert data as labels through a behavior cloning method;

constructing an initial expert decision network represented by a deep neural network;

and training the initial expert decision network by using the training sample set, and back-propagating a cross entropy loss function to update parameters of the initial expert decision network to obtain the expert decision network module.

4. A multifunctional radar interference decision method according to claim 3, characterized in that the expert data is represented in trajectory form as:

Γ＝{(s _i ,j _i )|i＝1,2,...,N _e }；

wherein s is _i Is the state of multifunctional radar, j _i For jammer pattern j _i E J, J is the interference pattern space,(s) _i ,j _i ) For the ith expert knowledge, N _e Representing the number of expert knowledge;

the cross entropy loss function is:

wherein M is the number of interference patterns, j _k For the kth interference pattern,probability distribution, p, of interference patterns assumed in the current multifunctional radar state _e (·|s) is the probability distribution of the interference pattern corresponding to the multi-functional radar state in the expert data.

5. The imperfect expert policy multifunction radar interference decision making method according to claim 1, wherein the interference decision making network comprises a main decision making network and a target decision making network, the main decision making network and the target decision making network are identical in structure, each comprising: a third fully-connected layer, a second active layer, a fourth fully-connected layer, a third active layer, a fifth fully-connected layer, a sixth fully-connected layer, and an addition module, wherein,

the third full-connection layer, the second activation layer, the fourth full-connection layer and the third activation layer are sequentially connected;

the input end of the fifth full-connection layer and the input end of the sixth full-connection layer are connected with the output end of the third full-connection layer;

the output end of the fifth full-connection layer and the output end of the sixth full-connection layer are both connected with the input end of the adding module;

the output end of the adding module is used as the output end of the main decision network or the target decision network.

6. The method for multi-functional radar interference decision making with imperfect expert strategy according to claim 5, wherein the trained decision network is obtained by training the decision network, and the training method of the decision network comprises the steps of:

s2041, setting the total number of training games, the maximum number of single-round games and an exploration factor, and initializing network parameters, learning rate, experience pool and sampling batch size of the interference decision network;

s2042, acquiring a current radar state, and inputting the current radar state into the expert intervention discriminant function module when the current radar state is inconsistent with the radar target state; when the current radar state is judged to be consistent with the radar target state, starting a new training round until the training times reach the total training game times;

s2043, when the expert intervention discriminant function module judges that the current radar state belongs to an expert decision error state set, selecting a current interference pattern by using a main decision network in the interference decision network to implement interference; when the expert interference judging function module judges that the current radar state does not belong to an expert decision error state set, judging the participation degree of an expert strategy for interference decision by utilizing the expert interference exploring function module according to the current radar state; when judging that the expert strategy participates in interference decision, selecting a current interference pattern by utilizing the expert decision network module to implement interference; when judging that the expert strategy does not participate in the interference decision, selecting a current interference pattern by using a main decision network in the interference decision network to implement interference;

s2044, acquiring a radar state after interference, evaluating current interference benefits by combining the current radar state, and storing the current radar state, the current interference pattern, the current interference benefits, the radar state after interference and a current ending signal as a combination into an experience pool;

s2045, sampling experience samples from the experience pool according to the size of the sampling batch by adopting a sampling mode of preferential experience playback so as to train and update the main decision network, and back-propagating by utilizing a target loss function so as to update main decision network parameters;

s2046, updating target decision network parameters by combining the super parameters and the main decision network parameters;

s2047, storing expert strategy error information into the expert decision error state set;

s2048, when the number of single-round games is smaller than the maximum number of single-round games, returning to the step S2042; and when the number of single-round games is equal to the maximum number of single-round games, returning to the step S2041 until the training number reaches the total number of training games, and obtaining the trained decision network.

7. The imperfect expert policy multi-functional radar interference decision making method according to claim 6, wherein the objective loss function is:

wherein D3QN is D3QN algorithm, θ is main decision network parameter, Q(s) _t ,j _t The method comprises the steps of carrying out a first treatment on the surface of the θ) is the primary decision network output at s _t Take j below _t Interference value s of (c) _t J is the current radar state _t R for the current interference pattern _t For current interference gain, Q _T For the target decision network s _t+1 For post-interference radar conditions, J is the interference pattern space, J is the interference pattern of the jammer, Q (s _t+1 J; θ) is the primary decision network output at s _t+1 The interference value of j, theta is adopted ^- Deciding network parameters for a target;

the update formula of the main decision network parameter is as follows:

wherein θ _new For updated primary decision network parameters, θ _old For the main decision network parameters before updating, l is the learning rate,regarding θ for the target loss function _old Is a gradient of (2);

the updating formula of the target decision network parameter is as follows:

wherein,decision network parameters for updated targets, +.>For the target decision network parameters before updating, τ is the superparameter.

8. The imperfect expert policy multi-functional radar interference decision making method according to claim 1, wherein the trained decision network output policy is:

wherein,for expert intervention discriminant function module->Exploring a function module for expert interference, pi _e For expert policy in expert decision network module, pi _q S is radar state, which is an interference strategy in the interference decision network.

9. The method of claim 1, wherein the expert intervention discriminant function module defines:

wherein s is the radar state,for expert decision error state set, < >>Representing the utilization of expert policy pi _e Decision making->Representing the utilization of interference policy pi _q And making a decision.

10. The method of claim 1, wherein the expert interference exploration function module defines:

wherein s is the radar state,for expert decision error state set, xi is random number of 0-1, epsilon _e In order to explore the factors,representing expert policy pi _e Participation in decision making, including->Representing the utilization of interference policy pi _q And (5) searching an interference pattern in the radar state s.