WO2023109640A1

WO2023109640A1 - Interpretability method and system for deep reinforcement learning model in driverless scene

Info

Publication number: WO2023109640A1
Application number: PCT/CN2022/137511
Authority: WO
Inventors: 周纪民; 刘延东; 张中劲; 王鲁佳; 王洋; 须成忠
Original assignee: 深圳先进技术研究院
Priority date: 2021-12-14
Filing date: 2022-12-08
Publication date: 2023-06-22
Also published as: CN114330109A

Abstract

The present invention relates to the field of model learning, and in particular to an interpretability method and system for a deep reinforcement learning model in a driverless scene. According to the method and system, a suitable simulation environment and a suitable deep reinforcement learning algorithm are selected to obtain a converged reinforcement learning model by training; a picture captured in a driverless scene is input into the reinforcement learning model, feature division is performed on the picture and quantitative analysis of feature influence is performed, the degrees of influence of features on model decision are calculated, a corresponding difference matrix is obtained, and an improved network model is obtained. Thus, the technical problem in the prior art that the influence of features of a picture on model decision cannot be accurately analyzed is at least solved.

Description

Method and system for interpretability of deep reinforcement learning model in unmanned driving scene

technical field

The invention belongs to the field of model learning, and in particular relates to an interpretability method and system for a deep reinforcement learning model in an unmanned driving scene.

Background technique

The interpretability technology of the deep reinforcement learning model in the unmanned driving scenario realizes the interpretation of the opaque model in the unmanned driving scenario, and has built-in deep reinforcement learning model problems and interpretation problems in the unmanned driving scenario. Solving algorithms and optimization schemes , which explains and visualizes to the user the important factors in the model's operation and decision-making process in an unmanned driving environment. Among them, the deep reinforcement learning model mainly involves the selection of various deep reinforcement learning algorithms (Deep Reinforcement Learning, DRL) for the autonomous decision-making of agents in an unmanned driving environment. As an emerging field of artificial intelligence, explainable artificial intelligence (ExplainableAI, XAI) is mainly to explain and visualize various AI algorithm models. Explainable reinforcement learning (Explainable Reinforcement Learning, XRL) is a branch of XAI technology, which explains the reinforcement learning model through a series of means, including the interpretation of the current deep reinforcement learning combined with deep learning, and the user can The understood text or image format is visualized and presented to the user.

DRL is an algorithm combining deep learning and reinforcement learning. It combines the perception ability of deep learning and the decision-making ability of reinforcement learning, and has been developed at a deeper level, including value-based DRL, policy-based (policy- based) DRL, model-based DRL, hierarchical-based DRL, etc.

XRL is interpretable reinforcement learning, which is used to explain and visualize DRL. XRL technologies are classified as follows:

(1) According to the time of extracting information, it is divided into: intrinsic interpretability (intrinsic interpretability), intrinsic interpretation is constructed to be intrinsically interpretable or itself interpretable during training, such as decision trees; post-hoc interpretability (post-hoc interpretability), post-hoc interpretation Alternative models or saliency maps are typical by providing an explanation for the original model after training by creating a second simpler model or other operations such as perturbation.

(2) According to the scope of explanation, it is divided into: global interpretability and local interpretability. Global interpretability explains the entire and general model behavior, while local interpretation provides explanations for specific decisions. Interpretability of global models is difficult to achieve in practice, especially for models beyond a few parameters. Thus local interpretability can be more easily applied. Explaining why a particular decision or single prediction was made means that interpretability happens locally. Typically, this approach to interpretability is used to generate individual explanations for why the model made a particular decision for an instance.

Interpretability technology of deep reinforcement learning models is a key issue in the field of unmanned driving and computers. As a subfield of XAI, XRL has not been widely studied. The current research direction of deep reinforcement learning is post-hoc interpretability, both global interpretability and local interpretability. Although XRL started late, there are currently several typical studies on post-hoc interpretability. At the same time, the explanatory algorithms for other artificial intelligence models in XAI can also be used in the interpretation of DRL.

Post-hoc interpretability: In 2018, Greydanus et al. proposed a salient map method (Saliencymap) on the ICML paper. This is a perturbation-based method that directly perturbs the input. Through the regional Gaussian blur, compare the difference between the normal picture and the blurred picture when passing through the network, and slide the blurred area on the picture to traverse the entire picture to obtain multiple differences. Among them, the area with a large difference plays an important role in the decision-making of the agent, and the key area of DRL learning is obtained. But existing perturbation-based methods for computing saliency often highlight regions of the input that are not relevant to the actions taken by the agent. In 2020, the method SARFA (Specific and Relevant Feature Attribution) proposed by Nikaash et al. in the ICLR paper generates more salient saliency maps by balancing two aspects (specificity and relevance), which capture Different salience needs. The first documents the effect of perturbations on the relative expected rewards of actions to be explained. The second part weighs irrelevant features that change the relative expected rewards of actions other than the one to be explained. It is also possible to approach the original black box model by training a second interpretable model. In 2016, Ribeiro et al. proposed the LIME algorithm in the SIGKDD paper, and trained a linear interpretable model to approximate the original classification network. CNN classification network is explained, and the method also uses the perturbation of the image.

Existing interpretability techniques for deep reinforcement learning models are based on perturbation techniques and training interpretable model approximations, but the methods currently proposed have limitations and great room for improvement, and cannot be quantified in the scope of interpretation Or the interpretation speed is low, and it cannot better provide reasonable explanations for deep reinforcement learning models. The limitations of existing algorithms cannot well meet the needs of practical applications for explanatory algorithms.

The saliency map algorithm (Saliencymap) explained by disturbing the input needs to perform regional Gaussian blurring on the input image at a certain interval, and input the blurred image into the network every time it is blurred, and the obtained value and the value obtained by entering the original image into the network Take the difference to get the degree of influence of the area on the model decision. In this way, it is not easy to obtain the influence of a specific feature in the picture on the model decision through uniform blurring. When the blur range is small, the entire feature cannot be covered, only a certain part of the feature Impact on decision-making; when the fuzzy range is relatively large, it is easy to cover multiple features, and the impact of a certain feature on model decision-making cannot be obtained, which is not conducive to accurate analysis of the impact of each feature of the image on model decision-making;

The LIME algorithm, which trains a simple model to approximate a complex model, uses a simple model to approximate a complex classification network, and uses a simple one-dimensional linear model to perform one-dimensional quantization and disturbance on the input image to approximate the original model. Finally, the model can be explained by looking at the magnitude of the coefficients of the linear model. This method can well explain the influence of the characteristics of the input image on the model's decision. However, LIME can only explain one sample at a time, and needs to build a new model each time. Although this algorithm is more general and accurate, it takes a long time to use, and it is difficult to use the data to update the network. And it is not very suitable for scenes with fast scene changes and high speed requirements.

technical problem

Embodiments of the present invention provide an interpretability method and system for a deep reinforcement learning model in an unmanned driving scene, to at least solve the technical problem that the prior art cannot accurately analyze the influence of each feature of a picture on model decision-making.

technical solution

According to an embodiment of the present invention, an interpretability method for a deep reinforcement learning model in an unmanned driving scene is provided, comprising the following steps:

Select a suitable simulation environment and a suitable deep reinforcement learning algorithm, and obtain a convergent reinforcement learning model through training;

Input the pictures taken in the unmanned driving scene to the reinforcement learning model, divide the features of the pictures and perform quantitative analysis of the influence of the features, calculate the degree of influence of each feature on the decision of the model, and obtain the corresponding difference matrix, and get the improved model network model.

Further, input the pictures taken in unmanned driving scenarios to the reinforcement learning model, divide the features of the pictures and perform quantitative analysis on the influence of features, calculate the degree of influence of each feature on the model decision, and obtain the corresponding difference matrix, The resulting improved network model includes:

First, the state image is obtained through the interaction between the model and the environment, and the image is divided into a fixed number of blocks according to the characteristics through superpixel segmentation, and the image set is obtained by sequentially blurring the Gaussian blur method of the irregular area;

Then input the image set and the original image into the network separately to obtain the decision value of the original image and the blurred image, and make a difference between the two to obtain the difference matrix;

The difference matrix is up-sampled so that the size of the matrix is equal to the size of the input image, and the value of the difference matrix is multiplied by a preset multiple and superimposed on the original image.

Furthermore, A3C in deep reinforcement learning is selected as the algorithm for the autonomous decision-making of the agent in unmanned driving.

Further, the unmanned driving environment selects the carla simulation environment, selects a suitable scene, and selects a picture as input.

Further, before inputting the pictures taken in the unmanned driving scene to the reinforcement learning model, it also includes: preprocessing the pictures taken in the unmanned driving scene.

Further, preprocessing the pictures taken in unmanned driving scenarios includes:

Convert the input image into the form required for interpretation: find and segment the appropriate image features in the unmanned driving environment, and use the minimum number of segmentation blocks to include the features required in the unmanned driving environment.

Further, image segmentation forms adjacent pixels with similar texture, color, and brightness characteristics into visually meaningful irregular pixel blocks, and replaces a large number of pixels with a small number of pixels; where image blurring is the average value of surrounding pixels for each pixel .

Further, use the saliency map algorithm to divide the features of the picture and perform quantitative analysis of the influence of the features.

Further, the method also includes:

Present the explained content to the user in a form that the user can easily understand.

According to another embodiment of the present invention, an interpretability system for a deep reinforcement learning model in an unmanned driving scenario is provided, including:

The network model module is used to select a suitable simulation environment and a suitable deep reinforcement learning algorithm, and obtain a convergent reinforcement learning model through training;

The explanatory algorithm module is used to input the pictures taken in the unmanned driving scene to the reinforcement learning model, divide the features of the pictures and perform quantitative analysis on the influence of features, calculate the degree of influence of each feature on the model decision, and obtain the corresponding The difference matrix is used to obtain an improved network model.

Beneficial effect

The interpretability method and system of the deep reinforcement learning model in the unmanned driving scene in the embodiment of the present invention, select a suitable simulation environment and a suitable depth reinforcement learning algorithm, obtain a convergent reinforcement learning model through training, and input the reinforcement learning model For the pictures taken in the unmanned driving scene, the features of the pictures are divided and the quantitative analysis of the influence of the features is carried out, the degree of influence of each feature on the model decision is calculated, and the corresponding difference matrix is obtained, and the improved network model is obtained.

Description of drawings

The accompanying drawings described here are used to provide a further understanding of the present invention and constitute a part of the application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations to the present invention. In the attached picture:

Fig. 1 is the frame diagram of the overall design of the interpretability method and system of the deep reinforcement learning model in the unmanned driving scene of the present invention;

Fig. 2 is a working flow chart of the interpretability method and system of the deep reinforcement learning model in the unmanned driving scene of the present invention.

Embodiments of the present invention

In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

The invention is a novel XRL algorithm, aiming at the deep reinforcement learning model, providing a fast and accurate solution for the interpretation and visualization of its decision-making; it quantifies the influence of the features determined in the input picture on the decision-making of the model; in order to improve Speed, reducing the number of superpixel blocks of each picture; in order to design a general interpretability algorithm that adapts to various actual scenarios, it does not depend on a specific model in the design process, so the XRL proposed by the present invention does not depend on A certain model (Model-free), and adapt to the actual scene problems, XRL should also have a certain degree of flexibility and scalability, so as to be able to adapt to various scenes with different numbers of features.

The problem to be solved by the present invention is to use algorithms such as Superpixel segmentation, Gaussian Blur, Saliencymap, and Deep reinforcement learning to solve the unknowable problem of deep reinforcement learning models in unmanned driving scenarios and Its extended problem enables users to understand the favorable and unfavorable factors in the decision-making process through explanatory algorithms, and presents them to users through good human-computer interaction. Present the decision-making basis of the intelligent body to the user, and increase the user's trust in the unmanned driving model.

The basic contents of the technical solution of the present invention include as follows:

1. Design a general-purpose explanatory algorithm, which is applicable to every series of algorithms in deep reinforcement learning;

2. For the design of input image preprocessing in the process of interpreting algorithms;

3. Interpretation of the decision-making impact through the network and pre-processed images;

4. Present the explanation content to the user through visualization technology.

The interpretability method and the overall system design framework of the deep reinforcement learning model in the unmanned driving scenario are composed of three parts: the network model part, the explanatory algorithm part, and the network improvement part, as shown in Figure 1.

(1) Network model part

The network model part includes the selection of deep reinforcement learning algorithms, scene design and model training in unmanned driving scenarios.

The present invention needs to select a suitable simulation environment and a suitable deep reinforcement learning algorithm in advance. Through comparison, A3C (Asynchronous advantageous actor-critic) in deep reinforcement learning is selected as the algorithm for the autonomous decision-making of the agent in unmanned driving. The unmanned driving environment selects the carla simulation environment, selects a suitable scene, and selects a picture as input: then through training, a convergent reinforcement learning model is finally obtained, and at this time, the model that is explained next is obtained.

(2) Explanatory algorithm part

The explanatory algorithm part includes image preprocessing, saliency map operation (solution of difference matrix), visualization and other modules.

Among them, the results of image preprocessing will be beneficial to the operation of the Saliencymap module and the division of features, and will be conducive to the quantitative analysis of the influence of explanatory algorithms on features; the Saliencymap module will calculate the impact of each feature on The degree of influence of model decision-making, and obtain the corresponding difference matrix, so as to obtain the important factors in the decision-making process of the model; the visualization module presents the explained content to users in a form that users can easily understand.

(3) Network improvement part

Through the explained information, strengthen the useful information, isolate the unimportant information, make the network effect better, and further verify the effect of the explanation.

Interpretability methods and systems for deep reinforcement learning models in unmanned driving scenarios need to meet three basic requirements:

(1) Unmanned driving scene. The scene is as rich as possible, close to the real situation, and a convergent deep reinforcement learning model is obtained.

(2) Interpretive algorithm, the preprocessing part tries to make the number of features separated from the picture appropriate.

(3) The area of the saliency map should be as convergent as possible, and not too scattered.

Based on these three requirements, the present invention designs the interpretability method and system workflow of the deep reinforcement learning model in the unmanned driving scene, as shown in FIG. 2 .

When the present invention obtains the required model, it starts to interpret the model. First, the state image is obtained through the interaction between the model and the environment, and the image is divided into a fixed number of blocks according to the characteristics through superpixel segmentation. The method of Gaussian Blur (GaussianBlur) in the regular area blurs the image set separately in turn. Afterwards, the image set and the original image are input into the network separately, so that the decision value of the original image and the blurred image are obtained, and the difference between the two is obtained to obtain the difference matrix. The difference matrix is up-sampled so that the size of the matrix is equal to the size of the input image, and the value of the difference matrix is multiplied by a certain multiple and superimposed on the original image, so as to be displayed to the user in the form of a saliency map. Afterwards, the area of the meaningful part is significantly enhanced, and the purpose of improving the network model is obtained.

Among them, the image preprocessing problem of XRL is the premise of XRL analysis. Converting the input image into the form required for the explanation of the present invention can be described as: find out the appropriate image features in the unmanned driving environment and segment them, which can be Some unimportant or relatively small features are not divided, and the minimum number of segmentation blocks is used to include the features required in the unmanned driving environment, thereby greatly reducing the time spent and achieving the desired effect. The preprocessing algorithm based on superpixel segmentation and Gaussian blur used in the present invention can well realize the segmentation of the main features of the input image and the Gaussian blur of irregular features, and is a better preprocessing method that can be used for deep reinforcement learning model interpretation .

Image preprocessing follows the traditional image processing algorithm process: image segmentation forms irregular pixel blocks with certain visual significance from adjacent pixels with similar texture, color, brightness and other characteristics, and replaces a large number of pixels with a small number of pixels. Image blurring can be understood as taking the average value of surrounding pixels for each pixel.

The feature of this explanatory algorithm is that the importance of features is quantified in each interpretation process, and there are positive and negative two for decision-making respectively. Therefore, only the parts that are different from traditional explanatory algorithms will be described:

Blur in irregular areas: The blurred part of the picture is to eliminate some features, so that the image is different from the original image in terms of features, which is convenient for subsequent comparison with the strategy obtained from the original image. Regarding the blurred parts, it should be noted that the blurred parts should be as far as possible Smooth transition with the unblurred part, so as not to affect the model's decision-making due to the obvious boundary between the blurred part and the unblurred part.

Feature interpretation: Process the picture according to the feature area, and obtain and quantify the influence of the feature area on the model decision-making. Through normalization processing, each feature is contrasted, and the positive contribution of each feature to the decision-making is calculated. Influence and negative influence, the obtained data is conducive to the next update of the model.

Improve the model based on the explanation: the obtained explanation can know which are the favorable factors and which are the unfavorable factors during the normal operation of the model. At the same time, when the model makes an error, it can also know which feature of the input image caused the failure of the system's decision-making. This information can be used to improve the model.

Key point of the present invention and want to protect point are at least:

1. The overall design of XRL;

2. Visualization of image preprocessing methods;

3. Improved algorithms for deep reinforcement learning models based on interpretability.

The present invention aims at the interpretability scene of the deep reinforcement learning model in the unmanned driving scene to solve the opaque problem of the DRL model in the scene, explain its decision-making, and provide a visualized human-computer interaction interface, which explains the depth to a certain extent. The interpretability of the reinforcement learning model increases the user's trust, and at the same time provides a basis for the improvement of the model. The present invention mainly embodies the following advantages:

Quantify and compare the impact of input image features on model decision-making to obtain important features;

Explain that the system is composed of modules and has the characteristics of high flexibility and good scalability;

In the image preprocessing stage, image features can be smoothed and blurred in irregular areas, so that the blurred area and the unblurred area can be smoothly handed over;

Deep reinforcement learning models can be further improved through the content of the explanation, which is currently not covered by explanation systems and model improvement systems.

Taking the unmanned driving environment as the experimental platform, the XRL and model improvement schemes are verified, and the XRL algorithm is verified. Through the visualization algorithm of the simulation platform, the user understands the decision-making basis of the model, and improves the model based on this basis.

The alternative scheme of the present invention is at least:

1. The XRL system is scalable, and expansion modules can be combined arbitrarily to meet customer needs. For example, adding or changing the image preprocessing process, changing the perturbation method of the image, changing the difference calculation method, etc.

2. Propose to improve the model through explanatory, enhance positive features, suppress negative features, and achieve model improvement.

The serial numbers of the above embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be realized in other ways. Wherein, the system embodiments described above are only illustrative, for example, the division of units can be divided into a logical function, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into Another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of units or modules may be in electrical or other forms.

A unit described as a separate component may or may not be physically separated, and a component shown as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed over multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server or a network device, etc.) execute all or part of the steps of the methods in various embodiments of the present invention. And the foregoing storage medium comprises: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

The above is only a preferred embodiment of the present invention, it should be pointed out that, for those of ordinary skill in the art, without departing from the principle of the present invention, some improvements and modifications can also be made, and these improvements and modifications can also be made. It should be regarded as the protection scope of the present invention.

Claims

An interpretability method for a deep reinforcement learning model in an unmanned driving scene, characterized in that it comprises the following steps:

Select a suitable simulation environment and a suitable deep reinforcement learning algorithm, and obtain a convergent reinforcement learning model through training;

Input the pictures taken in the unmanned driving scene to the reinforcement learning model, divide the features of the pictures and perform quantitative analysis of the influence of the features, calculate the degree of influence of each feature on the decision of the model, and obtain the corresponding difference matrix, and get the improved model network model.
The interpretability method of the deep reinforcement learning model under the unmanned driving scene according to claim 1, characterized in that, input the picture taken under the unmanned driving scene to the reinforcement learning model, and divide the features of the picture and perform feature influence Quantitative analysis of power, calculate the degree of influence of each feature on model decision-making, and obtain the corresponding difference matrix, the improved network model includes:

First, the state image is obtained through the interaction between the model and the environment, and the image is divided into a fixed number of blocks according to the characteristics through superpixel segmentation, and the image set is obtained by sequentially blurring the Gaussian blur method of the irregular area;

Then input the image set and the original image into the network separately to obtain the decision value of the original image and the blurred image, and make a difference between the two to obtain the difference matrix;

The difference matrix is up-sampled so that the size of the matrix is equal to the size of the input image, and the value of the difference matrix is multiplied by a preset multiple and superimposed on the original image.
The interpretability method of a deep reinforcement learning model in an unmanned driving scene according to claim 1, wherein A3C in deep reinforcement learning is selected as an algorithm for autonomous decision-making of an agent in unmanned driving.
The interpretability method of the deep reinforcement learning model under the unmanned driving scene according to claim 1, characterized in that, the unmanned driving environment selects a carla simulation environment, selects a suitable scene, and selects a picture as input.
The interpretability method of the deep reinforcement learning model under the unmanned driving scene according to claim 1, wherein, before inputting the picture taken under the unmanned driving scene to the reinforcement learning model, it also includes: taking pictures under the unmanned driving scene images are preprocessed.
The interpretability method of the deep reinforcement learning model under the unmanned driving scene according to claim 5, wherein the preprocessing of the pictures taken under the unmanned driving scene comprises:

Convert the input image into the form required for interpretation: find and segment the appropriate image features in the unmanned driving environment, and use the minimum number of segmentation blocks to include the features required in the unmanned driving environment.
According to claim 6, the interpretability method of deep reinforcement learning model under unmanned driving scene is characterized in that, image segmentation forms adjacent pixels with similar texture, color and brightness characteristics into irregular pixel blocks with visual significance , and replace a large number of pixels with a small number of pixels; where image blurring takes the average value of surrounding pixels for each pixel.
The interpretability method of a deep reinforcement learning model in an unmanned driving scene according to claim 1, wherein a saliency map algorithm is used to divide the features of the picture and perform a quantitative analysis of the influence of the features.
The interpretability method of the deep reinforcement learning model under the unmanned driving scene according to claim 1, it is characterized in that, described method also comprises:

Present the explained content to the user in a form that the user can easily understand.
An interpretability system for a deep reinforcement learning model in an unmanned driving scenario, characterized in that it includes:

The network model module is used to select a suitable simulation environment and a suitable deep reinforcement learning algorithm, and obtain a convergent reinforcement learning model through training;

The explanatory algorithm module is used to input the pictures taken in the unmanned driving scene to the reinforcement learning model, divide the features of the pictures and perform quantitative analysis on the influence of features, calculate the degree of influence of each feature on the model decision, and obtain the corresponding The difference matrix is used to obtain an improved network model.