CN105388461A - Radar adaptive behavior Q learning method - Google Patents

Abstract

The invention belongs to the field of radar signal processing, and particularly relates to a Q learning method updated based on a Bayesian table to learn and recognize radar adaptive behaviors. The invention provides a radar adaptive behavior Q learning method. An improved Q learning algorithm is used for learning in view of a time domain waveform selection behavior (the minimum mutual information criterion), a big forward step is made on the basis of carrying out interference only according to direct information obtained by a receiving end traditionally, a suggested machine learning algorithm is used for recognizing the radar time domain adaptive behavior, and a certain learning result is given. The method of the invention applies the Q learning algorithm updated based on the Bayesian table to the radar behavior learning and recognition problem for the first time, and in comparison with the prior art, learning effects under time domain waveform selection (the minimum mutual information criterion) are better.

Description

A Radar Adaptive Behavior Q Learning Method

technical field

The invention belongs to the field of radar signal processing, and in particular relates to the problem of learning and identifying radar adaptive behavior by a Q learning method based on Bayesian table updating.

Background technique

With the advent of the adaptive system and adaptive signal processing in the 1960s, the adaptive radar system was born, and its adaptive capability is developing day by day, and gradually developed from the radar receiver adaptive processing to the receiver-transmitter synchronous processing . At present, the adaptive behavior of radar is mainly manifested in the behavioral characteristics of time/frequency/space domain working parameters, signal processing and working mode, such as adaptive behavior of time domain waveform selection. Radar waveform selection is an important means of radar waveform self-adaptation. The target radar will build a waveform library, and select transmit waveforms in the waveform library according to certain criteria to improve radar performance. The waveform selection criterion is closely related to the working mode (or radar task) of the radar. According to the existing literature, the waveform selection criterion includes the maximum mutual information criterion when the radar task is target identification (the target is the next transmission signal and the target maximum best matching); the minimum amount of mutual information (the target is that the signal sent next time can obtain more new information amount, and the common amount of information is minimum); the maximum Kullback-Leibler information criterion, etc., the present invention will take the minimum amount of mutual information as the object .

In the current field of radar signal processing, as the interference party, it generally identifies fixed radar targets. However, intelligence is a trend in future development. Both sides will gradually develop in the direction of cognitive ability, and have adaptive The goal of behavior requires more intelligent algorithms to learn adaptive behavior, and then use the learning results to carry out efficient and real-time attacks.

Q-learning algorithm is a kind of reinforcement learning algorithm, which was first proposed by C.Watkins in his doctoral dissertation "Learning from delayed rewards" in 1989. This algorithm is a powerful combination of dynamic programming theory and animal learning psychology to solve Sequential decision problems with delayed rewards are the goal. In the Q learning algorithm, the behavior value function of the Markov decision process is iteratively calculated according to the time difference, and the iterative calculation formula is: $Q({\mathrm{the\; s}}_t,a_t)\←Q({\mathrm{the\; s}}_t,a_t)+\mathrm{\α}\[r_{t+1}+\mathrm{\γ}\underset{a\∈A_{\mathrm{the\; s}}}{max}Q({\mathrm{the\; s}}_{t+1},a_t)-Q({\mathrm{the\; s}}_t,a_t)\],$ Among them, the parameter α is called the learning rate (or learning step size), and γ is the discount rate. Q(s _t , a _t ) is the value function of the state-action pair, which means that in the state s _t , execute the action a _t , and then map the action according to the strategy π. The reward obtained, the goal of Q learning is Every step it takes is greedy.

Bayesian network represents knowledge by providing a graphical method. It is a directed acyclic graph, in which nodes represent variables in the domain of discourse, directed arcs represent the relationship between variables, and conditional probability represents the degree of influence between variables. Bayesian networks can clearly reflect the dependencies between variables in practical applications. Bayesian network, also known as belief network, is a graphical model that represents the joint probability distribution function between a set of variables. A Bayesian network consists of a structural model and a set of conditional probability distribution functions associated with it.

When the number of data features in the Bayesian network is K, then the joint distribution p(x ₁ ,...,x _K ) of these K variables can be written as the following form, and through the Bayesian network Simplify with conditionally independent features: $\begin{array}{c}p(x_1,...,x_K)=p\left(x_K\right|x_1,...,x_{K-1})...p\left(x_2\right|x_1)p\left(x_1\right)\\ =\underset{k=1}{\overset{K}{\Π}}p\left(x_k\right|{\mathrm{pa}}_k)\end{array},$ Among them, pa _k refers to the set of parent nodes of node x _k . It can be seen that when the Bayesian network is relatively sparse, the form of the joint probability density will be greatly simplified.

Q-learning is an unsupervised machine learning algorithm. Through learning, the learner can gradually adapt to the environment to be learned. Here, it refers to the adaptive behavior of adapting to the target radar, while Bayesian learning converts the unknown from the perspective of probability (confidence). As a random variable, information has good adaptability and scalability. Applying Bayesian learning theory to Q learning and adding heuristic strategies can achieve better results in the learning of adaptive behavior of target radar.

Contents of the invention

The purpose of the present invention is to aim at the deficiencies in the prior art, provide a kind of radar self-adaptive behavior Q learning method, use the improved Q learning algorithm to learn for the time-domain waveform selection behavior (minimum mutual information criterion), in the tradition only according to receiving Based on the interference of the direct information obtained from the terminal, a big leap forward is made, and the proposed machine learning algorithm is used to identify the adaptive behavior of the radar in the time domain, and a certain learning result is given.

The technical solution of the present invention is: on the basis of the Q-learning algorithm, the waveform selection under the time-domain minimum mutual information criterion is used as the learning object. First, the waveform database information of the target radar waveform needs to be obtained before the adaptive behavior learning of the target radar waveform ;Secondly, model the adaptive behavior object, and use the modeled object to interact with the learning party to obtain the waveform transition under different disturbances, that is, the laboratory training data; then, use the training data to learn the parameters of the Bayesian network , to obtain the Bayesian posterior probability table and the Bayesian record table; finally, using the Bayesian posterior probability table as prior knowledge, use the proposed algorithm for iterative learning, and give the learning results

A radar adaptive behavior Q learning method, the specific steps are as follows:

S1. The learning party forces the target radar to change the transmission signal by continuously transmitting the detection interference signal, and the receiving end of the learning party obtains the next transmission signal of the target radar, which is used to improve the dynamic waveform library of the learning party. The dynamic waveform library of the sound learning party The specific method is: compare the waveform information obtained by the receiving end of the learner with the known waveform, if there is no such waveform in the dynamic waveform library, store it in the dynamic waveform library, and then continue to transmit the detection interference signal until m times of interaction get The target radar emission waveforms can be found in the dynamic waveform library, where m is the empirical value;

S2. Select the waveform under the minimum mutual information criterion in the time domain as the learning object, model it, and use the modeled object to interact with the learning party to obtain the waveform transition under different interferences, that is, the laboratory training data;

S3, use the training data described in S2 to learn Bayesian network parameters, use the Bayesian toolbox under the Matlab environment, add Dirichlet prior distribution, and obtain the maximum posterior probability table of the new radar waveform, that is, Bayesian A recording table, wherein the Bayesian recording table refers to the new waveform number with the maximum posterior probability under the existing waveform and the existing interference signal;

S4. On the basis of the original Q-learning algorithm, take the Bayesian record table described in S3 as prior knowledge, perform iterative learning according to the Bayesian table updating algorithm, and give the learning result.

Further, the specific method of modeling described in S2 is:

S21. When the target radar waveform selection criterion is the minimum mutual information criterion, the radar echo signal is modeled as b=a+w=Sα+w, wherein, S is a waveform convolution matrix containing waveform parameters, and α is a scattering coefficient vector , w is the receiver noise vector;

S22. Perform waveform selection, specifically: ensure that the waveform to be sent next time can obtain more new information, that is, the mutual information of the two radar echo signals before and after is the smallest, that is $\begin{array}{c}mI=\left\{mI(b_1,b_i)\right\}_{{\mathrm{the\; s}}_i\left(m\right)}^{\min }\\ mI(b_1,b_i)=h\left(b_1\right)-h\left(b_1|b_i\right)\\ =h\left(b_i\right)-h\left(b_i|b_1\right)\end{array},$ Under the assumption of w Gaussian white noise distribution, the amount of mutual information between waveform 1 and waveform i is, Among them, {d _k |k=1,2,...,K} is the singular value of the cross-correlation matrix R _xz , and the matrix R _xz is defined as The singular value satisfies: 1≥d ₁ ≥d ₂ ...≥d _K ≥0, the cross-correlation matrix R ₁₁ , R _i1 , and R _ii are defined as $\begin{array}{c}\mathrm{E.}\left(b_1{b}_1^h\right)=R_{11}=S_1{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h+R_{ww}\\ \mathrm{E.}\left(b_i{b}_1^h\right)=R_{i1}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h\\ \mathrm{E.}\left(b_i{b}_i^h\right)=R_{ii}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_i^h+R_{ww}\end{array},$ Get the amount of mutual information between different waveforms;

S23, modeling the waveform selection object described in S22, characterizing different radar waveform states with parameters such as signal waveform radar and bandwidth, and selecting the waveform with the minimum mutual information with the previous transmitted waveform in the waveform library as a new radar waveform state;

S24. Set different interference signals to affect the waveform selection of the target radar, and continuously interact with each other to obtain the waveform transitions under different interferences, that is, laboratory training data.

Further, the Bayesian network parameter learning described in S3 is specifically:

S31. Using the training data described in S2 to obtain the conditional probability and Bayesian theorem in the Bayesian network;

S32. Obtain the posterior probability of the output node, namely the root node, according to the conditional probability and Bayesian theorem described in S31 Among them, s _k refers to the state of the radar at time k, r _k refers to the attack taken by the learning party at time k, s _k+1 refers to the new state of the radar at time k+1, and the left side of the formula indicates that the radar is in state s _{k at time k} , learning When the party attacks r _k , the probability that the radar changes to the new state sk+1 at time _k+1 is the posterior probability estimate of the new state of the radar. In the denominator on the right side of the formula, P(s _k+1 |s _k ) represents the state transition probability of the radar, which is also the prior probability of the state at time k+1, and P(r _k |s _k+1 ,s _k ) is the radar state Conditional probability, which means that the radar is in state s _k at time k, and the state s k+1 at time _k+1 , the probability that the learner takes an action r _k , that is, in state s _k , set a desired state s _k+1 , the learning party chooses the probability of each attack to make the radar change from state s _k to state s _k+1 , and the denominator P(r _k |s _k ) is the integral or summation of the numerator to the new state s _k+1 , Still taking the current state s _k as the condition, the learning party chooses the probability of attacking r _k .

Further, the Bayesian table update algorithm described in S4 is as follows:

S41. After performing the waveform selection object modeling under the minimum mutual information amount, radar waveform library waveform parameter setting, and interference signal parameter setting, the waveform conversion situation is obtained by transmitting different interference signals to the target radar, that is, the laboratory training data is obtained. The specific implementation process is as follows: waveform selection starts from waveform 1, and the attack randomly selects from 4 interference numbers to obtain a new waveform, update and loop, and obtain 100 training data;

S42. Construct a Bayesian network, add the prior distribution, solve the maximum posterior solution, and use the Bayesian toolbox in Matlab to analyze the conditional probability in the Bayesian network Solve, and finally get the posterior probability of the root node, where the prior probability is set to Dirichlet distribution, the probability is equal, and the Bayesian record table is the waveform under different disturbances that is counted on the basis of the solved maximum posterior probability solution For the transfer situation, for the current waveform and a certain interference, select the new waveform with the largest posterior probability as the output, and record it in the table, that is, $S_{t+1}^{\max }=\underset{S_{t+1}}{\arg \max }\left\{P(S_t,r_t,S_{t+1})\right\};$

S43. After obtaining the Bayesian posterior probability table, update the algorithm flow chart according to the Bayesian table, interact with the waveform selection object under the minimum mutual information criterion, and then iterate and learn the algorithm.

The beneficial effects of the present invention are:

The method of the invention applies the Q-learning algorithm based on Bayesian table update to the radar behavior learning and identification problem for the first time, and the learning effect under the time-domain waveform selection (minimum mutual information criterion) is better than that of the prior art.

Description of drawings

Figure 1 is a schematic diagram of the method for obtaining the target radar waveform library.

Fig. 2 is a schematic diagram of waveform adaptive behavior modeling.

Figure 3 is the Bayesian network structure under the waveform selection object.

Figure 4 is a flowchart of the Q-learning algorithm.

Fig. 5 is a flow chart of Bayesian table update algorithm.

Fig. 6 is the convergence curve of the Q learning algorithm.

Fig. 7 is the convergence curve of the Bayesian table update algorithm.

Fig. 8 is a state transition diagram under the interference policy verification after learning.

Figure 9 is the verification of the learning effect of the algorithm when the initial waveforms are different.

detailed description

The technical solution of the present invention will be described in detail below in combination with the embodiments and the accompanying drawings.

S1. The learning party forces the target radar to change the transmission signal by continuously transmitting and detecting interference signals, and the receiving end of the learning party obtains the next transmission signal of the target radar, and continuously improves the dynamic waveform library of the learning party. First, compare the waveform information obtained by the receiving end of the learning party with the known waveform. If the waveform does not exist in the dynamic waveform library, store it in the dynamic waveform library, and then continue to transmit and detect interference until the target radar transmission waveform obtained in m interactions All can be found in our waveform library, and the value of m can be adjusted.

As shown in Figure 1, taking the minimum mutual information criterion under waveform selection as the object, the target radar waveform library is continuously transmitted to traverse the target radar waveform library, and the learning square waveform library is obtained. The subsequent learning and simulation have completely and correctly traversed the target radar waveform. carried out under the conditions of the library.

S2. Select the waveform under the minimum mutual information criterion in the time domain as the learning object, model it, and use the modeled object to interact with the learning party to obtain the waveform transition under different interferences, that is, the laboratory training data. details as follows:

S21. Perform characteristic analysis and modeling on the waveform adaptive behavior.

When the target radar waveform selection criterion is the minimum mutual information criterion, the radar echo signal is modeled as: b=a+w=Sα+w, where S is the waveform convolution matrix containing waveform parameters, and α is the scattering coefficient vector, w is the receiver noise vector, which is generally assumed to be Gaussian white noise.

Radar uses more efficient methods of gathering information to accurately describe areas of interest. Therefore, the waveform selection criterion is to ensure that the waveform sent next time can obtain more new information, that is, the mutual information of the two radar echo signals before and after is the smallest, and the expression is: $\begin{array}{c}mI=\left\{mI(b_1,b_i)\right\}_{{\mathrm{the\; s}}_i\left(m\right)}^{\min }\\ mI(b_1,b_i)=h\left(b_1\right)-h\left(b_1|b_i\right)\\ =h\left(b_i\right)-h\left(b_i|b_1\right)\end{array}.$

Under the assumption of w Gaussian white noise distribution, the mutual information between waveform 1 and waveform i is: Among them, {d _k |k=1,2,...,K} is the singular value of the cross-correlation matrix R _xz , and the matrix R _xz is defined as follows: The singular value satisfies: 1≥d ₁ ≥d ₂ ...≥d _K ≥0, the cross-correlation matrix R ₁₁ , R _i1 , and R _ii are defined as: $\begin{array}{c}\mathrm{E.}\left(b_1{b}_1^h\right)=R_{11}=S_1{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h+R_{ww}\\ \mathrm{E.}\left(b_i{b}_1^h\right)=R_{i1}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h\\ \mathrm{E.}\left(b_i{b}_i^h\right)=R_{ii}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_i^h+R_{ww}\end{array},$ From the above formula, the mutual information between different waveforms can be obtained.

Then, the waveform selection object under the minimum mutual information criterion is modeled, and different radar waveform states are represented by parameters such as signal waveform radar and bandwidth, and the waveform library with the minimum mutual information with the previous transmitted waveform is selected according to this criterion. The waveform is used as a new radar waveform state, therefore, the transition of the state under a certain criterion is the radar waveform adaptive behavior.

The specific modeling is shown in Figure 2. A radar waveform state is characterized by parameters such as signal waveform type, signal bandwidth, and signal pulse width. In the simulation of the present invention, the target radar waveform parameters are set as follows: 32 waveforms and 8 types of waveforms are set in the waveform library, which are respectively linear frequency modulation positive slope, linear frequency modulation negative slope, secondary frequency modulation concave positive slope, and secondary frequency modulation concave negative slope , Secondary FM downward convex positive slope, secondary FM downward convex negative slope, logarithmic FM positive slope, logarithmic FM negative slope; each type of waveform can be set with 4 bandwidths, 10MHz, 15MHz, 20MHz, 25MHz.

After modeling the waveform selection object under the minimum mutual information criterion, it is also necessary to set the interference signal parameters of the learning side. There are 4 types of interference signals, which are single-frequency signals with an interference-to-signal ratio of 30dB and 50dB, and the interference-to-signal ratio is 30dB and a 55dB chirp signal with a bandwidth of 30MHz.

After having the learning object and the interference signal, it is necessary to solve the mutual information between the waveforms under different interferences in the simulation process in order to obtain the newly selected waveform. Among them, under the assumption of w Gaussian white noise distribution, the mutual information between waveform 1 and waveform i is: Among them, {d _k |k=1,2,...,K} is the singular value of the cross-correlation matrix R _xz , and the matrix R _xz is defined as follows: The singular value satisfies: 1≥d ₁ ≥d ₂ ...≥d _K ≥0, the cross-correlation matrix R ₁₁ , R _i1 , and R _ii are defined as: $\begin{array}{c}\mathrm{E.}\left(b_1{b}_1^h\right)=R_{11}=S_1{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h+R_{ww}\\ \mathrm{E.}\left(b_i{b}_1^h\right)=R_{i1}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h\\ \mathrm{E.}\left(b_i{b}_i^h\right)=R_{ii}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_i^h+R_{ww}\end{array},$ From the above formula, the mutual information between different waveforms can be obtained.

S22. After modeling the minimum mutual information waveform selection object, set different interference signals to affect the waveform selection of the target radar, so as to continuously interact, and then obtain the waveform transformation under different interferences, that is, the laboratory training data. The radar waveform parameters and interference signal parameters during the interaction process are described in detail below.

S3. Use the training data to learn the parameters of the Bayesian network, use the Bayesian toolbox under the Matlab environment, add the Dirichlet prior distribution, and obtain the maximum posterior probability table of the new radar waveform, and the Bayesian record table It refers to the number of the new waveform with the maximum posterior probability under the current waveform and the current interference signal.

The structure of the Bayesian network is shown in Figure 3. The training data is used to obtain various conditional probabilities and Bayesian theorem in the Bayesian network, and then the posterior probability of the output node, that is, the root node, can be obtained. Among them, s _k refers to the state of the radar at time k, r _k refers to the attack taken by the learning party at time k, s _k+1 refers to the new state of the radar at time k+1, and the left side of the formula indicates that the radar is in state s _{k at time k} , learning When the party attacks r _k , the probability that the radar changes to the new state sk+1 at time _k+1 is the posterior probability estimate of the new state of the radar. In the denominator on the right side of the formula, P(s _k+1 |s _k ) represents the state transition probability of the radar, which is also the prior probability of the state at time k+1, and P(r _k |s _k+1 ,s _k ) is the radar state Conditional probability, which means that the radar is in state s _k at time k, and the state s k+1 at time _k+1 , the probability that the learner takes an action r _k , that is, in state s _k , set a desired state s _k+1 , the learning party chooses the probability of each attack to make the radar change from state s _k to state s _k+1 , and the denominator P(r _k |s _k ) is the integral or summation of the numerator to the new state s _k+1 , Still taking the current state s _k as the condition, the learning party chooses the probability of attacking r _k .

S4. On the basis of the original Q-learning algorithm, using the Bayesian posterior probability table as prior knowledge, perform iterative learning according to the flow chart of the Bayesian table updating algorithm in Figure 5, and give the learning results.

Figure 4 and Figure 5 are the flow charts of the Q-learning algorithm and the Q-learning algorithm based on Bayesian table updating, respectively. The main difference between the two algorithms is that the Bayesian table updating algorithm uses the laboratory training data to obtain the Bayesian posterior Probability table, and use this table as prior knowledge and guided knowledge of the target waveform, and then learn and iterate during the interaction with the object.

The specific implementation process of the Q-learning algorithm includes the following steps:

Step 1. After the above-mentioned modeling of the waveform selection object under the minimum amount of mutual information, radar waveform library waveform parameter setting, and interference signal parameter setting, the Q learning algorithm can be used to interact with the target object by using the interference signal.

Step 2. According to the flow chart of the Q-learning algorithm in FIG. 4, the Q-learning algorithm iterates and learns during different interactions with the target object. FIG. 6 shows the convergence curve of the Q-learning algorithm. Among them, the number of episodes on the abscissa indicates the number of times the target state is reached, and the number of iterations per episode on the ordinate indicates the number of attacks required to reach the target state each time, that is, when the learner interacts with the target radar, the target radar The number of interactions required to reach the goal state. It can be seen from the figure that at the beginning of the simulation sequence, the number of iterations required is very large, and even the upper limit of the number of iterations will be reached. As the simulation sequence deepens, based on the knowledge obtained during the previous iterations , the number of iterations required by the algorithm to reach the target waveform gradually decreases, and finally reaches stability.

The specific implementation process of the Bayesian table update algorithm includes the following steps:

Step 1. After the above-mentioned modeling of the waveform selection object under the minimum mutual information, radar waveform library waveform parameter setting, and interference signal parameter setting, the waveform conversion situation can be obtained by transmitting different interference signals to the target radar, that is, The laboratory training data is obtained, and the specific implementation process is as follows: waveform selection starts from waveform 1, and the attack randomly selects from 4 interference numbers to obtain a new waveform, update and loop, and obtain 100 training data.

Step 2: Construct a Bayesian network, as shown in Figure 3, add the prior distribution, and find the maximum a posteriori solution. The prior probability is set to Dirichlet distribution with equal probability. Use the Bayesian toolbox in Matlab to solve the conditional probability in the Bayesian network according to the following formula, and finally get the posterior probability of the root node.

The Bayesian recording table is the waveform transition under different disturbances calculated on the basis of the maximum a posteriori probability solution that has been solved. Among them, for the current waveform and a certain disturbance, select a new waveform with the maximum a posteriori probability As output, i.e. recorded in the table, i.e. $S_{t+1}^{\max }=\underset{S_{t+1}}{\arg \max }\left\{P(S_t,r_t,S_{t+1})\right\}.$

Step 3: After obtaining the Bayesian posterior probability table, update the algorithm flowchart according to the Bayesian table in Figure 5, and interact with the waveform selection object under the minimum mutual information criterion, and then the algorithm iterates and learns , while Figure 7 is the algorithm convergence curve, and Figure 8 and Figure 9 are the learning results of the Bayesian table update algorithm.

The convergence of the Q learning algorithm and the convergence of the Bayesian table update algorithm can be seen in Figure 6 and Figure 7, as can be seen, the Q learning algorithm based on the Bayesian table update proposed by the present invention has better convergence, that is, for The learning effect of radar waveform adaptive behavior is better; after Bayesian table update algorithm learning, it can be seen from the statistical curve of the number of iterations in Figure 7 that the number of iterations gradually decreases until it stabilizes, and when it reaches stability, the algorithm has completed the learning of the target object , followed by the battlefield verification stage, using the Bayesian posterior probability table obtained iteratively in the laboratory learning stage and the interference selection strategy of the algorithm, under the same initial state and target state, the interference pattern selection and state transition on the battlefield As shown in Figure 8; when the initial value is not waveform 1, the average number of iterations required to reach the target waveform under different initial waveforms can be obtained by using the posterior probability table learned by the Bayesian table update algorithm, as shown in the figure 9. The abscissa indicates the initial waveform number initialized for each iteration, and the ordinate indicates the number of algorithm iterations when the target waveform is reached under the corresponding initial waveform. It can be seen that after iterative learning, the number of attacks required to reach the target waveform under each initial waveform is greatly reduced, all within 10 times.

Claims (4)

1. a radar adaptive behavior Q learning method, is characterized in that, comprises the steps: S1. The learning party forces the target radar to change the transmission signal by continuously transmitting the detection interference signal, and the receiving end of the learning party obtains the next transmission signal of the target radar, which is used to improve the dynamic waveform library of the learning party. The dynamic waveform library of the sound learning party The specific method is: compare the waveform information obtained by the receiving end of the learner with the known waveform, if there is no such waveform in the dynamic waveform library, store it in the dynamic waveform library, and then continue to transmit the detection interference signal until m times of interaction get The target radar emission waveforms can be found in the dynamic waveform library, where m is the empirical value; S2. Select the waveform under the minimum mutual information criterion in the time domain as the learning object, model it, and use the modeled object to interact with the learning party to obtain the waveform transition under different interferences, that is, the laboratory training data; S3, use the training data described in S2 to learn Bayesian network parameters, use the Bayesian toolbox under the Matlab environment, add Dirichlet prior distribution, and obtain the maximum posterior probability table of the new radar waveform, that is, Bayesian A recording table, wherein the Bayesian recording table refers to the new waveform number with the maximum posterior probability under the existing waveform and the existing interference signal; S4. On the basis of the original Q-learning algorithm, take the Bayesian record table described in S3 as prior knowledge, perform iterative learning according to the Bayesian table updating algorithm, and give the learning result. 2. A kind of radar adaptive behavior Q learning method according to claim 1, is characterized in that: the specific method of modeling described in S2 is: S21. When the target radar waveform selection criterion is the minimum mutual information criterion, the radar echo signal is modeled as b=a+w=Sα+w, wherein, S is a waveform convolution matrix containing waveform parameters, and α is a scattering coefficient vector , w is the receiver noise vector; S22. Perform waveform selection, specifically: ensure that the waveform to be sent next time can obtain more new information, that is, the mutual information of the two radar echo signals before and after is the smallest, that is $\begin{array}{c}mI=\underset{{\mathrm{the\; s}}_i\left(\mathrm{no}\right)}{\min }\left\{mI\left(b_1,b_i\right)\right\}\\ mI\left(b_1,b_i\right)=h\left(b_1\right)-h\left(b_1|b_i\right)\\ =h\left(b_i\right)-h\left(b_i|b_1\right)\end{array},$ Under the assumption of w Gaussian white noise distribution, the amount of mutual information between waveform 1 and waveform i is, Among them, {d _k |k=1,2,...,K} is the singular value of the cross-correlation matrix R _xz , and the matrix R _xz is defined as The singular value satisfies: 1≥d ₁ ≥d ₂ ...≥d _K ≥0, the cross-correlation matrix R ₁₁ , R _i1 , and R _ii are defined as $\begin{array}{c}\mathrm{E.}\[b_1{b}_1^h\]=R_{11}=S_1{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h+R_{ww}\\ \mathrm{E.}\[b_i{b}_1^h\]=R_{i1}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_1^h\\ \mathrm{E.}\[b_i{b}_i^h\]=R_{ii}=S_i{R}_{\mathrm{\α}\mathrm{\α}}{S}_i^h+R_{ww}\end{array},$ Get the amount of mutual information between different waveforms; S23, modeling the waveform selection object described in S22, characterizing different radar waveform states with parameters such as signal waveform radar and bandwidth, and selecting the waveform with the minimum mutual information with the previous transmitted waveform in the waveform library as a new radar waveform state; S24. Set different interference signals to affect the waveform selection of the target radar, and continuously interact with each other to obtain the waveform transitions under different interferences, that is, laboratory training data. 3. a kind of radar adaptive behavior Q learning method according to claim 1, is characterized in that: the Bayesian network parameter learning described in S3 is specifically: S31. Using the training data described in S2 to obtain the conditional probability and Bayesian theorem in the Bayesian network; S32. Obtain the posterior probability of the output node, namely the root node, according to the conditional probability and Bayesian theorem described in S31 Among them, s _k refers to the state of the radar at time k, r _k refers to the attack taken by the learning party at time k, s _k+1 refers to the new state of the radar at time k+1, and the left side of the formula indicates that the radar is in state s _{k at time k} , learning When the party attacks r _k , the probability that the radar changes to the new state sk+1 at time _k+1 is the posterior probability estimate of the new state of the radar. In the denominator on the right side of the formula, P(s _k+1 |s _k ) represents the state transition probability of the radar, which is also the prior probability of the state at time k+1, and P(r _k |s _k+1 ,s _k ) is the radar state Conditional probability, which means that the radar is in state s _k at time k, and the state s k+1 at time _k+1 , the probability that the learner takes an action r _k , that is, in state s _k , set a desired state s _k+1 , the learning party chooses the probability of each attack to make the radar change from state s _k to state s _k+1 , and the denominator P(r _k |s _k ) is the integral or summation of the numerator to the new state s _k+1 , Still taking the current state s _k as the condition, the learning party chooses the probability of attacking r _k . 4. a kind of radar adaptive behavior Q learning method according to claim 1, is characterized in that: Bayesian table update algorithm described in S4, specifically as follows: S41. After performing the waveform selection object modeling under the minimum mutual information amount, radar waveform library waveform parameter setting, and interference signal parameter setting, the waveform conversion situation is obtained by transmitting different interference signals to the target radar, that is, the laboratory training data is obtained. The specific implementation process is as follows: waveform selection starts from waveform 1, and the attack randomly selects from 4 interference numbers to obtain a new waveform, update and loop, and obtain 100 training data; S42. Construct a Bayesian network, add the prior distribution, solve the maximum posterior solution, and use the Bayesian toolbox in Matlab to analyze the conditional probability in the Bayesian network Solve, and finally get the posterior probability of the root node, where the prior probability is set to Dirichlet distribution, the probability is equal, and the Bayesian record table is the waveform under different disturbances that is counted on the basis of the solved maximum posterior probability solution For the transfer situation, for the current waveform and a certain interference, select the new waveform with the largest posterior probability as the output, and record it in the table, that is, $S_{t+1}^{\max }=\underset{S_{t+1}}{\arg \max }\left\{P(S_t,r_t,S_{t+1})\right\};$ S43. After obtaining the Bayesian posterior probability table, update the algorithm flow chart according to the Bayesian table, interact with the waveform selection object under the minimum mutual information criterion, and then iterate and learn the algorithm.