CN113283516A - Multi-sensor data fusion method based on reinforcement learning and D-S evidence theory - Google Patents

Abstract

The invention discloses a multi-sensor data fusion method based on reinforcement learning and a D-S evidence theory, which comprises the following steps: step one, inputting data of multiple sensors; step two, constructing a Markov decision model; thirdly, the Q-learning algorithm realizes conflict data resolution; and step four, realizing multi-sensor data fusion by adopting a D-S evidence theory. The method based on the combination of reinforcement learning and D-S evidence theory is adopted to quickly and efficiently realize the multi-sensor data fusion, and the reinforcement learning is adopted to perform real-time online processing on conflict data and fault data in the multi-sensor to obtain effective multi-sensor data after conflict resolution, so that the problem of high data conflict is solved; and secondly, uncertain multi-source data can be fused well by adopting a D-S evidence theory fusion method, and the fusion performance is improved.

Description

A Multi-sensor Data Fusion Method Based on Reinforcement Learning and D-S Evidence Theory

technical field

The invention relates to the field of multi-sensor data fusion, which is a method for realizing multi-sensor data fusion based on reinforcement learning and D-S evidence theory, and realizes real-time online and effective fusion of multi-sensor data.

Background technique

With the application of modern science and technology in industrial equipment, the equipment is complicated, and the information of a single sensor can no longer accurately reflect the complex operation of the equipment. And because it is easily disturbed by the environment, the obtained data may contain faulty data, which cannot realize accurate decision-making on complex equipment. The multi-sensor can simultaneously reflect the different characteristics of the system, and the fusion of multi-source data is beneficial to improve the performance of the system and make the results more credible.

Multi-sensor data fusion technology is an important data processing technology that makes full use of multi-sensor data in different time and space to analyze, sort and fuse the data to improve system performance, and plays a key role in actual production and application.

Among multi-sensor data fusion models and methods, D-S evidence theory is one of the most effective methods to deal with uncertain information. Evidence theory also provides the Dempster combination rule, which enables evidence fusion without prior information. However, counterintuitive judgments arise when there is a high degree of conflict between the evidence. In addition, in practice, the data obtained by multiple sensors may have time inconsistency, which makes this method unable to realize real-time fusion of online data. Reinforcement learning learns by "trial and error", which can realize autonomous real-time interaction with the external environment without system prior information. Therefore, the combination of reinforcement learning and D-S evidence theory is used to realize real-time online fusion of multi-sensor data.

SUMMARY OF THE INVENTION

In order to realize multi-sensor real-time online data fusion, the present invention provides an intelligent multi-sensor data fusion method based on reinforcement learning and D-S evidence theory, which solves the problem of conflicting evidence and online real-time fusion.

The technical scheme adopted by the present invention to solve its technical problems comprises the following steps:

Step 1: Multi-sensor data input

The acquired multi-sensor data is expressed as: {D ₁ , D ₂ ,...,D _i }, where D _i represents the data of the ith sensor, and there are i sensors in total in this system. In addition, the data expression form of the sensor is Basic Probability Assignment (BPA). If new sensor data is added in the measurement process, it will be written into the data, expressed as: {D ₁ , D ₂ ,..., D _i , D _new }.

Step 2: Construction of Markov Decision Model

When using reinforcement learning for adaptive conflict resolution of multi-sensor data, a Markov decision process (MDP) model needs to be constructed. The MDP model includes system states, actions, and reward functions, specifically:

(1) Action set A: Due to the different amount of information of multi-source sensors, the system should make different action choices under different sensor information, so that when there is high conflict information, the conflict resolution is performed on the high conflict information to ensure fusion. The accuracy and validity of the result, so the action set is defined as: A={a ₁ , a ₂ }={reservation, deletion}, the system can take the action of retaining or deleting according to the actual situation;

其中m_t和m_t+1表示当前时刻采取不同动作的融合结果，a_t+1表示当前采取的动作。系统状态集合表示为：S＝{s₁,s₂,…,s_t,…}。(2) State set S: When the fusion system takes a certain action under different sensor information at different times, the state of the system will transfer. We define the fusion result at the current moment as the state of the system, namely:

Among them, m _t and m _t+1 represent the fusion result of different actions taken at the current moment, and a _t+1 represents the currently taken action. The system state set is expressed as: S={s ₁ ,s ₂ ,...,s _t ,...}.

(3) Reward function R: Represents the reward value or penalty value given by the system under a certain state s and a certain action a during the operation of the system.

In the multi-sensor data fusion algorithm, different actions are selected through the setting of the reward function value, and finally a more accurate and effective fusion result is obtained under the same data condition. In reinforcement learning, the optimal action is obtained by maximizing the accumulated reward value, so the setting of the reward function is very important. In the present invention, the reward function is set according to the quality of the fusion result in a certain state, and Deng entropy is used to evaluate the quality of the fusion result, and Deng entropy is a measure of reliability between evidences. The Deng entropy E(m _t ) in different states of the system is defined as:

Among them, Θ represents the recognition frame, A is the subset in the recognition frame, the corresponding BPA at time t is m _t (A), and |A| represents the potential of A.

The larger the value of Deng entropy, the larger the amount of information contained in the BPA, indicating that the fusion result in the current state is better. When E(m _t+1 )≥E(m _t ), it means that the new state s _t+1 is beneficial, and a positive reward should be given at this time. On the contrary, it means that the new state s _t+1 is negative, and a penalty value should be given at this time. Therefore, the reward function at time t+1 is defined as:

Step 3: Q-learning algorithm realizes conflict data resolution

The purpose of realizing conflict resolution in multi-sensor data fusion is to find an optimal strategy π: π:S→A, that is, a=π(s), so that the system can make the best strategy choice in the case of conflicting data and achieve effective Conflict resolution. The selection of the strategy is made by the environment and the agent through repeated exploration and trial and error, and finally, under a certain strategy, the strategy with the largest sum of the system's immediate reward and future reward value is the optimal strategy. Specifically: at time t, the multi-sensor data fusion system receives a certain state s _t and an immediate reward R _t , determines the current action to be performed (retains or removes the current evidence) according to the reward value, and then the intelligent fusion system transfers to the next A state s _t+1 produces a reward R _t+1 . The present invention adopts the Q-learning algorithm to realize this process.

The Q-learning algorithm obtains the best strategy by maximizing the cumulative discount reward value. The Q value function is the cumulative reward value under a certain state s _t and a certain action a _t , and the expression is:

where γ represents the discount factor.

The ε-greedy algorithm is used to select the optimal action (strategy), which is specifically expressed as:

Among them, π ^* (a|s) represents the optimal strategy; ε represents the exploration probability.

And use the following formula to update the Q-value function:

where α∈(0,1] represents the learning rate and s _t+1 represents the next state.

The multi-sensor data fusion system obtains the optimal action at each moment through repeated updating and interaction with the environment, and finally eliminates the high-conflict BPA to realize adaptive online data processing.

Step 4: Using evidence theory to realize multi-sensor data fusion

After the conflict resolution processing of the sensor data, the Dempster combination rule in the evidence theory is used to realize the effective fusion of multi-source data, which is as follows:

in,

The beneficial effect of the present invention is that the present invention adopts the method of combining evidence theory and reinforcement learning, which can efficiently and quickly realize the fusion of multi-sensor data; The digested effective multi-sensor data solves the problem of high data conflict; the D-S evidence theory fusion method adopted in the present invention can well fuse uncertain multi-source data and improve fusion performance.

Description of drawings

Fig. 1 is the overall model diagram that the present invention realizes;

Figure 2 is the car failure data;

Figure 3 is the result of multi-sensor data fusion.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and examples. An example of fault diagnosis of automobile system is given here, and the experimental data comes from [1]. It is shown in [1] that there are three types of faults in the automotive system (represented here by F ₁ , F ₂ and F ₃ ), which are low oil pressure, air leakage in the intake system, and solenoid valve stuck. In addition, it obtains vehicle fault data through five sensors C ₁ , C ₂ , C ₃ , C ₄ , and C ₅ . The implementation steps of the proposed method are described in the present invention in conjunction with the vehicle system fault data.

Step 1: Multi-sensor data input

The multi-sensor data obtained from the automotive system is represented as: {D ₁ , D ₂ , D ₃ , D ₄ , D ₅ }, indicating that there are a total of 5 sensors in the system. In the fault data of the automobile system, the _C5 sensor is in a failed state, and the rest of the sensors are working normally. The specific data are shown in Figure 2. Among them, m(F ₁ ) represents the reliability value of fault type F ₁ , m(F ₂ ) represents the reliability value of fault type F ₂ , m(F ₃ ) represents the reliability value of fault type F ₃ , m(Θ) represents the reliability value for which the fault type cannot be determined, Θ={F ₁ , F ₂ , F ₃ }. The purpose of the present invention is to determine which type of failure has occurred in the vehicle system based on multi-sensor data.

Step 2: Construction of Markov Decision Model

Reinforcement learning is used for the adaptive conflict resolution of the vehicle system data, and the Markov decision process (MDP) model is constructed first. The MDP model includes system states, actions, and reward functions, specifically:

(1) Action set A: Due to the different information amounts of the five sensors, the system should make different action choices under different sensor information, so that when there is high-conflict information, the conflict resolution is performed on the high-conflict information to ensure fusion. The accuracy and validity of the result, so the action set is: A={a ₁ , a ₂ }={retain, delete}.

其中m_t和m_t+1表示当前时刻采取不同动作的融合结果，a_t+1表示当前采取的动作。系统状态集合表示为：S＝{s₁,s₂,…,s_t,…}。(2) State set S: when the fusion system takes a certain action, the state of the system will transition. The present invention defines the fusion result at the current moment as the state of the system, namely:

Among them, m _t and m _t+1 represent the fusion result of different actions taken at the current moment, and a _t+1 represents the currently taken action. The system state set is expressed as: S={s ₁ ,s ₂ ,...,s _t ,...}.

(3) Reward function R: Represents the reward value or penalty value given by the system under a certain state s and a certain action a during the operation of the system. In the present invention, the reward function is set according to the quality of the fusion result in a certain state, and Deng entropy is used to evaluate the quality of the fusion result, and Deng entropy is a measure of reliability between evidences. The Deng entropy E(m _t ) in different states of the system is defined as:

Among them, Θ represents the recognition frame, A is the subset in the recognition frame, the corresponding BPA at time t is m _t (A), and |A| represents the potential of A.

The larger the value of Deng entropy, the larger the amount of information contained in the BPA, indicating that the fusion result in the current state is better. When E(m _t+1 )≥E(m _t ), it means that the new state s _t+1 is beneficial, and a positive reward should be given at this time. On the contrary, it means that the new state s _t+1 is negative, and a penalty value should be given at this time. Therefore, the reward function at time t+1 is defined as:

Step 3: Q-learning algorithm realizes conflict data resolution

The purpose of realizing conflict resolution in multi-sensor data fusion is to find an optimal strategy π: π:S→A, that is, a=π(s), so that the system can make the best strategy choice in the case of conflicting data and achieve effective Conflict resolution. The selection of the strategy is made by the environment and the agent through repeated exploration and trial and error, and finally, under a certain strategy, the strategy with the largest sum of the system's immediate reward and future reward value is the optimal strategy. Specifically: at time t, the multi-sensor data fusion system receives a certain state s _t and an immediate reward R _t , determines the current action to be performed (retains or removes the current evidence) according to the reward value, and then the intelligent fusion system transfers to the next A state s _t+1 produces a reward R _t+1 . The present invention adopts the Q-learning algorithm to realize this process.

The Q-learning algorithm obtains the best strategy by maximizing the cumulative discount reward value. The Q value function is the cumulative reward value under a certain state s _t and a certain action a _t , and the expression is:

where γ represents the discount factor.

The ε-greedy algorithm is used to select the optimal action (strategy), which is specifically expressed as:

Among them, π ^* (a|s) represents the optimal strategy; ε represents the exploration probability.

And use the following formula to update the Q-value function:

where α∈(0,1] represents the learning rate and s _t+1 represents the next state.

The multi-sensor data fusion system obtains the optimal action at each moment through repeated updating and interaction with the environment, and finally eliminates the high-conflict BPA to realize adaptive online data processing.

Step 4: Use D-S evidence theory to realize multi-sensor data fusion

After the conflict resolution processing of the sensor data, the Dempster combination rule in the evidence theory is used to realize the effective fusion of multi-source data, which is as follows:

in,

Finally, the multi-sensor data fusion method based on reinforcement learning and D-S evidence theory of the present invention is adopted, and the simulation analysis is performed according to the data in Figure 2, and the simulation is compared with the traditional Dempster combination rule and Yager combination rule. The fusion result is shown in Figure 3. It can be seen from FIG. 3 that the fusion method proposed by the present invention has relatively high fusion accuracy, and can realize efficient, accurate and intelligent fusion under multi-sensor data. The main reason is that reinforcement learning eliminates conflicting data and fault data in multi-sensor data, and avoids such data from negatively affecting fusion results. In addition, the advantage of this method is that it does not need to consider whether the data arrival time is consistent, which can effectively realize the online fusion of data.

The basic principles and main features of the present invention and the advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments, and the descriptions in the above-mentioned embodiments and the description are only to illustrate the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have Various changes and modifications fall within the scope of the claimed invention. The claimed scope of the present invention is defined by the appended claims and their equivalents.

references

[1] Yan X H, Zhu J H, Kuang M C, et al. Missile aerodynamic design using reinforcement learning and transfer learning. Sci China Inf Sci, 2018, 61: 119204.

Claims (1)

1. A multi-sensor data fusion method based on reinforcement learning and D-S evidence theory is characterized by comprising the following steps:

the method comprises the following steps: multi-sensor data input

The multi-sensor data obtained is represented as: { D₁,D₂,…,D_iIn which D is_iData representing the ith sensor, and there are i sensors in the system. In addition, the data expression of the sensor is in the form of a Basic Probability Assignment function (BPA); if new sensor data is added during the measurement process, the new sensor data is written into the data, and the data is expressed as: { D₁,D₂,…,D_i,D_New}；

Step two: markov decision model construction

When the reinforcement learning is adopted to perform self-adaptive conflict resolution on multi-sensor data, firstly, a Markov Decision Process (MDP) model is constructed, wherein the MDP model comprises a system state, an action and a reward function, and specifically comprises the following steps:

(1) movable partMaking a set A: define the action set as: a ═ a₁,a₂The system can take the actions of retention or deletion according to the actual situation;

(2) a state set S: the invention defines the fusion result of the current moment as the state of the system, namely:

wherein m is_tAnd m_t+1Indicating a fusion result of taking different actions at the current moment, a_t+1Representing the currently taken action, the system state set is represented as: s ═ S₁,s₂,…,s_t,s_t+1,…}；

(3) The reward function R: the invention sets a reward function according to the quality of the fusion result in a certain state, adopts the Deng entropy to evaluate the quality of the fusion result, and adopts the Deng entropy E (m) of the system in different states_t) Is defined as:

wherein, Θ represents the recognition framework, a is a subset in the recognition framework, and the corresponding BPA at time t is m_t(A) And | a | represents the potential of a;

the larger the value of the Dune entropy is, the larger the information content contained in the BPA is, which indicates that the fusion result in the current state is better; when E (m)_t+1)≥E(m_t) Now, the new state s is explained_t+1Advantageously, a positive reward should be given; conversely, a penalty value should be given; the reward function at time t +1 is thus defined as:

step three: q-learning algorithm for realizing conflict data resolution

The Q-learning algorithm achieves the best strategy by maximizing the cumulative discount reward value, the Q value function being at a certain states_tAnd a certain action a_tThe following jackpot value is expressed as:

wherein γ represents a discount factor;

the method adopts an epsilon-greedy algorithm to select the optimal action (strategy), and is specifically represented as follows:

wherein, pi^*(a | s) represents an optimal strategy; epsilon represents the exploration probability;

and updating the Q value function by adopting the following formula:

wherein, alpha is (0, 1)]Indicates the learning rate, s_t+1Represents the next state;

the multi-sensor data fusion system obtains the optimal action at each moment through repeated updating interaction with the environment, and finally eliminates high-conflict BPA to realize self-adaptive online data processing;

step four: multi-sensor data fusion realized by adopting D-S evidence theory

After conflict resolution processing is carried out on sensor data, effective fusion of multi-source data is realized by adopting Dempster combination rules in evidence theory, and the method specifically comprises the following steps:

wherein,

Images