US10872187B2 - Verified runtime validation of verified cyber-physical system models - Google Patents

Abstract

A method for ensuring that verification results about models apply to cyber-physical systems (CPS) implementations is presented. The invention provides correctness guarantees for CPS executions at runtime. Offline verification of CPS models are combined with runtime validation of system executions for compliance with the model. The invention ensures that the verification results obtained for the model apply to the actual system runs by monitoring the behavior of the world for compliance with the model, assuming the system dynamics deviation is bounded. If, at some point, the observed behavior no longer complies with the model, such that offline verification results no longer apply, provably safe fallback actions are initiated. The invention includes a systematic technique to synthesize provably correct monitors automatically from CPS proofs in differential dynamic logic.

Description

RELATED APPLICATIONS

This application claims the benefit, under 35 U.S.C.

371, of PCT Application No. PCT/US14/60019, filed Oct. 10, 2014, which claims the benefit of U.S. Provisional Application No. 61/961,366, filed on Oct. 11, 2013, and U.S. Provisional Application No. 61/999,318, Filed Jul. 23, 2014. All of these applications are hereby incorporated by reference.

GOVERNMENT INTEREST

This invention was made with government support under DARPA Number FA87501220291. The government has certain rights in this invention.

FIELD OF THE INVENTION

This invention pertains to the field of cyber-physical systems (CPS), and, in particular to models approximating the behavior of CPS and methods for monitoring the actual CPS that are based on the models.

BACKGROUND OF THE INVENTION

Formal verification and validation play a crucial role in making cyber-physical systems (CPS) safe. Formal methods make strong guarantees about the system behavior if accurate models of the system can be obtained, including models of the controller and of the physical dynamics. In CPS, models are essential; but any model that can be built necessarily deviates from the real world. If the real system fits to the model, the behavior of the real system can be guaranteed to satisfy the correctness properties verified with respect to the model. If not, then this verification is impossible.

Cyber-physical systems (“CPS”) span controllers and the relevant dynamics of the environment. Since safety is crucial for CPS, their models (e. g., hybrid system models) need to be verified formally. Formal verification guarantees that a model is safe with respect to a safety property. The remaining task is to validate whether the models are adequate, such that the verification results transfer to the system implementation.

Actual system execution, however, provides many opportunities for surprising deviations from the model. Faults may cause the system to function improperly, sensors may deliver uncertain values and actuators may suffer from disturbance. The formal verification may have assumed simpler ideal-world dynamics for tractability reasons or made un-realistically strong assumptions about the behavior of other agents in the environment. Simpler models are often better for real-time decisions and optimizations, because they make predictions feasible to compute at the required rate. The same phenomenon of simplicity for predictability is often exploited for the models in formal verification and validation. As a consequence, the verification results obtained about models of a CPS only apply to the actual CPS at runtime to the extent that the system fits to the model.

Validation, i.e., checking whether a CPS implementation fits to a model, is an interesting but difficult problem. CPS models are more difficult to analyze than ordinary (discrete) programs because of the physical plant, the environment, sensor inaccuracies, and actuator disturbance. In CPS, models are essential; but any model necessarily deviates from the real world. Still, good models can be correct within certain error margins.

SUMMARY OF THE INVENTION

Herein we introduce a method (referred to herein as “ModelPlex”) to synthesize monitors by theorem proving. The method uses sound proof rules to formally verify that a model is safe and to synthesize provably correct monitors that validate compliance of system executions with the model.

This invention performs runtime model validation. That is, validating whether a model, shown for example, in FIG. 3 as reference number 100, which is assumed for verification purposes is adequate for a particular system execution to ensure that the verification results apply to the current execution. The invention checks system execution with respect to a monitor specification 104, and thus, belongs to the field of runtime verification. Herein, the term runtime validation is used to clearly convey the purpose of monitoring. While “runtime verification” monitors properties without offline verification, “runtime validation” monitors model adequacy to transfer offline verification results. The focus of the invention is on verifiably correct runtime validation to ensure that verified properties of models provably apply, which is important for safety and certification of CPS.

If an observed system execution fits to the verified model 12, then this execution is safe according to the offline verification result about the model. If it does not fit, then the system is potentially unsafe because it no longer has an applicable safety proof, so a verified fail-safe action to avoid safety risks is initiated. Checking whether a system execution fits to a verified model 12 includes checking that the actions chosen by the (unverified) controller implementation fit to one of the choices and requirements of the verified controller model. It also includes checking that the observed states can be explained by the plant model. The crucial questions are: How can a compliance monitor be synthesized that provably represents the verified model? How much safety margin does a system need to ensure that fail-safe actions are initiated early enough for the system to remain safe even if its behavior ceases to comply with the model?

The second question is related to feedback control and can only be answered when assuming constraints on the deviation of the real system dynamics from the plant model. Otherwise, if the real system can be infinitely far off from the model, safety guarantees are impossible. By the sampling theorem in signal processing, such constraints further enable compliance monitoring solely on the basis of sample points instead of the unobservable intermediate states about which no sensor data exists. When such constraints are not available, the method of the present invention still generates verifiably correct runtime tests, which detect deviation from the model at the sampling points, not just between them. A fail-safe action will then lead to best-effort mitigation of safety risks (rather than guaranteed safety).

As presented herein, the present invention is a method to synthesize verifiably correct runtime validation monitors automatically. ModelPlex uses theorem proving with sound proof rules to turn hybrid system models into monitors in a verifiably correct way. Upon non-compliance with the model, ModelPlex initiates provably safe fail-safe actions.

ModelPlex is a principle to build and verify high-assurance controllers for safety-critical computerized systems that interact physically with their environment. It guarantees that verification results about CPS models transfer to the real system by safeguarding against deviations from the verified model. Monitors created by ModelPlex are provably correct and check at runtime whether or not the actual behavior of a CPS complies with the verified model and its assumptions. Upon non-compliance, ModelPlex initiates fail-safe fallback strategies. To initiate the fallback strategies early enough, ModelPlex uses prediction on the basis of disturbed plant models to check safety for the next control cycle. This way, ModelPlex ensures that verification results about a model of a CPS transfer to the actual system behavior at runtime.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows the use of ModelPlex monitors along a system execution.

FIG. 2 show the water tank simulation using monitors

FIG. 3 is a high level flow chart of the process of deriving monitors from a model.

FIG. 4 is a flow chart of the “Analyze Monitor Specification” step of the process shown in FIG. 3

DETAILED DESCRIPTION OF THE INVENTION Preliminaries: Differential Dynamic Logic

For hybrid systems verification differential dynamic logic

is used, which has a notation for hybrid systems as hybrid programs.

allows us to make statements that we want to be true for all runs of a hybrid program ([α]ø) or for at least one run

α

ø) of a hybrid program.

Both constructs are necessary to derive safe monitors: [α]ø proofs are needed to ensure that all behavior of a model (including controllers) is safe;

α

ø proofs are needed to find monitor specifications that detect whether or not system execution fits to the verified model.

Table 1 summarizes the relevant syntax fragment of hybrid programs together with an informal semantics. The semantics ρ(α) of hybrid program α is a relation on initial and final states of running α. The set of

formulas is generated by the following grammar (<∈{<, ≤, =, ≥, >}, and θ₁, θ₂ are arithmetic expressions in +, −, ·, / over the reals):
Φ::=θ₁˜θ₂|

Φ|Φ

ψ|Φ

ψ|Φ→ψ|∀xΦ|∃xΦ|[α]Φ|

α

Φ

Differential dynamic logic comes uses a theorem prover with a verification technique to prove correctness properties of hybrid programs. One example of a theorem prover and the one that is used in the preferred embodiment of the invention is KeYmaera.

TABLE 1
Hybrid program representations of hybrid systems.

Statement	Effect
α; β	sequential composition, first run hybrid program α, then
	hybrid program β
α ∪ β	nondeterministic choice, following either hybrid
	program α or β
α*	nondeterministic repetition, repeats hybrid program
	α n ≥ 0 times
x := θ	assign value of term θ to variable x (discrete jump)
x := *	assign arbitrary real number to variable x
?F	check that a particular condition F holds, and abort if it
	does not
(x1′ = θ1, . . . ,	evolve xi along differential equation system xi′ = θi
xn′ = θn & F)	restricted to maximum evolution domain F

ModelPlex Approach for Verified Runtime Validation

CPS are almost impossible to get right without sufficient attention to prior analysis, for instance by formal verification and formal validation techniques. We assume to be given a verified model of a CPS, i.e. formula (1) is proved valid. Note that differential dynamic logic (

) and KeYmaera as a theorem prover are used to illustrate the concepts herein. The concept of ModelPlex is not predicated on the use of KeYmaera to prove (1). Other verification techniques could be used to establish validity of this formula. The flexibility of the underlying logic

, its support for both [α]Φ and

α

Φ, and its proof calculus, however, are exploited for systematically constructing monitors from proofs in the sequel.
Φ→[α*]ψ with invariant φ→[α]φs.t.Φ→φ and φ→ψ  (1)

Formula (1) expresses that all runs of the hybrid system α*, which start in states that satisfy the precondition ø and repeat the model α arbitrarily many times, must end in states that satisfy the post condition ψ. Formula (1) is proved using some form of induction, which shows that a loop invariant φ holds after every run of α if it was true before. The model α is a hybrid system model 100 of a CPS, which means that it describes both the discrete control actions of the controllers in the system and the continuous physics of the plant and the system's environment.

The safety guarantees obtained by proving formula (1) about the model α* transfer to the real system, if the actual CPS execution fits to α*. Because safety properties should be preserved, a CPS γ fits to a model α*, if the CPS reaches at most those states that are reachable by the model, i.e., ρ(γ)⊂ρ(α*). However, we do not know γ and therefore need to find a condition based on α* that can be checked at runtime to see if concrete runs of γ behave like α*.

Checking the post condition ψ is not sufficient because, if ψ does not hold, the system is already unsafe. Checking the invariant φ is insufficient as well, because if φ does not hold, the controller can no longer guarantee safety, even though the system may not yet be unsafe. But if we detect when a CPS is about to deviate from α* before leaving φ, we can still switch to a fail-safe controller to avoid

ψ from happening.

As shown in FIG. 1, ModelPlex derives three kinds of monitors, a model monitor 20, a controller monitor 22, and a prediction monitor 24. Reachability between consecutive states in α, α_ctrl, and α_δplant are checked by verifying states during execution against the corresponding monitor.

Model monitor 20—In each state v_i the sample point v_i−1 from the previous execution γ_i−1 is tested for deviation from the single α, not α* i. e., test (v_i−1, v_i) ∈ρ(α). If violated, other verified properties may no longer hold for the system. The system, however, is still safe if a prediction monitor 24 was satisfied on v_i−1.Frequent violations indicate an inadequate model that should be revised to better reflect reality.

Controller monitor 22—In intermediate state {tilde over (v)}_i the current controller decisions of the implementation γ_ctrl are tested for compliance with the model, i.e., test (v_i, {tilde over (v)}_i) ∈ρ(α_ctrl). Controller monitors 22 are designed for switching between controllers similar to Simplex. If violated, the commands from a fail-safe controller replace the current controller's decisions to ensure that no unsafe commands are ever actuated.

Prediction monitor 24—In intermediate state {tilde over (v)}_i the worst-case safety impact of the current controller decisions are tested with respect to the predictions of a bounded deviation plant model α_δplant, which has a tolerance around the model plant α_plant, i.e., check v_i+1|=φ for all v_i+1 such that ({tilde over (v)}_i, v_i+1) ∈ρ(α_δplant). Note, that all v_i+1 are simultaneously checked by checking {tilde over (v)}_i for a characterizing condition of α_δplant. If violated, the current control choice is not guaranteed to keep the system safe until the next control cycle and, thus, a fail-safe controller takes over.

The assumption for the prediction monitor 24 is that the real execution is not arbitrarily far off the plant models used for safety verification, because otherwise guarantees can neither be made on unobservable intermediate states nor on safety of the future system evolution. A separation of disturbance causes in the models can be used: ideal plant models α_plant for correctness verification purposes, implementation deviation plant models α_δplant for monitoring purposes. Any deviation model (e.g., piecewise constant disturbance, differential inclusion models of disturbance) is supported, as long as the deviation is bounded and differential invariants can be found. It is assumed that monitor evaluations are at most some ε time units apart (e.g., along with a recurring controller execution). Note that disturbance in α_δplant is more manageable compared to α*, as single runs α can be focused on instead of repetitions for monitoring.

Relation Between States

A check that inspects states of the actual CPS to detect deviation from the model α* is systematically derived. First, a notion of state recall is established and shows that, when all previous state pairs comply with the model, compliance of the entire execution can be checked by checking the latest two states (v_i−1, v_i).

Definition 1 (State recall). V is used to denote the set of variables whose state we want to recall. Y_V ⁻≡

_x∈V ^x=x− are used to express a characterization of the values of variables in a state prior to a run of α, where the fresh variables x⁻ to are always presumed to occur solely in Y_V ⁻. The variables in x⁻ can be used to recall this state. Likewise, Y_V ⁺≡

_x∈V ^x=x+ is used to characterize the posterior states and expect fresh x⁺.

With this notation, the following lemma states that an interconnected sequence of α transitions forms a transition of α*.

Lemma 1 (Loop prior and posterior state). Let α be a hybrid program and α* be the program that repeats arbitrarily many times. Assume that all consecutive pairs of states (v_i−1, v_i) ∈ρ(α) of n ∈

⁺ executions, whose valuations are recalled with Y_V ⁱ≡

_x∈Vx=xⁱ and Y_V ⁱ⁻¹ are plausible with respect to the model α, i.e., . |=

_1≤i≤n (Y_V ⁱ⁻¹→

α

Y_V ⁱ) with Y_V ⁻=Y_V ⁰ and Y_V ⁺=Y_V ⁿ. Then, the sequence of states originates from an α* execution from Y_V ⁰ to Y_V ⁿ, i.e., |=Y_V ^31→

^α*

^Y _V ⁺.

Lemma 1 enables us to check compliance with the model α* up to the current state by checking reachability of a posterior state from a prior state on each execution of α (i.e., online monitoring, which is easier because the loop was eliminated). To find compliance checks systematically, we construct formula (2), which relates a prior state of a CPS to its posterior state through at least one path through the model α. Consecutive states for α* mean before and after executions of α (i.e., α;↓α;↓α, not within α).
Y_V ⁻→(α)Y_V ⁺  (2)

Formula (2) is satisfied in a state v, if there is at least one run of the model α starting in the state v recalled by Y_V ⁻ and which results in a state ω recalled using Y_V ⁺. In other words, at least one path through α explains how the prior state v got transformed into the posterior state ω. The

formula (2) characterizes the state transition relation of the model α directly. Its violation witnesses compliance violation. Compliance at all intermediate states cannot be observed by real-world sensors. (See Section ‘Monitoring Compliance Guarantees for Unobservable Intermediate States’ herein).

In principle, formula (2) would be a monitor, because it relates a prior state to a posterior state through the model of a CPS, but the formula is difficult, if not impossible, to evaluate at runtime, because it refers to a hybrid system α, which includes non-determinism and differential equations. The basic observation is that any formula that is equivalent to (2) but conceptually easier to evaluate in a state would be a correct monitor 112. We use theorem proving for simplifying formula (2) into quantifier-free first-order real arithmetic form so that it can be evaluated efficiently at runtime. The resulting first-order real arithmetic formula can be easily implemented in a runtime monitor and executed along with the actual controller. A monitor 112 is executable code that only returns true if the transition from the prior system state to the posterior state is compliant with the model 100. Thus, deviations from the model 100 can be detected at runtime, so that appropriate fallback and mitigation strategies can be initiated.

Remark 1. The complexity for evaluating an arithmetic formula over the reals for concrete numbers is linear in the formula size, as opposed to deciding the validity of such formulas, which is doubly exponential. Evaluating the same formula on floating point numbers is inexpensive, but may yield wrong results due to rounding errors; on exact rationals, the bit-complexity can be non-negligible. Interval arithmetic is used to obtain reliable results efficiently.

Example 1

A simple water tank is used as a running example to illustrate the concepts throughout this section. The water tank has a current level x and a maximum level m. The water tank controller, which runs at least every ε time units, non-deterministically chooses any flow f between a maximum outflow −1 and a maximum

$\frac{m-x}{\varepsilon }.$
This water tank never overflows, as witnessed by a proof for the following

formula.

$\underset{\varphi }{\underbrace{0\le x\le m\bigwedge \varepsilon >0}}\to \hspace{1em}\left[{\left(f\text{:=}*;?\left(-1\le f\le \frac{m-x}{\varepsilon }\right);t\text{:=}0;\left(x^{\prime }=f,t^{\prime }=1\&x\ge 0\bigwedge t\le \varepsilon \right)\right)}^{*}\right]\stackrel{\underset{︷}{\varphi }}{\left(0\le x\le m\right)}$

ModelPlex Monitor Synthesis

This section introduces the nature of ModelPlex monitor specifications, the approach in this invention for generating such specifications from hybrid system models, and how to turn those specifications into monitor code that can be executed at runtime along with the controller.

A ModelPlex specification 104 corresponds to the

formula (2). If the current state of a system does not satisfy a ModelPlex specification 104, some behavior that is not reflected in the model 100 occurred (e.g., the wrong control action was taken, unanticipated dynamics in the environment occurred, sensor uncertainty led to unexpected values, or the system was applied outside the specified operating environment).

A model monitor X_m checks that two consecutive states

and

can be explained by an execution of the model α, i.e., (

,

) ∈ρ(α). In the sequel, BV(α) are bound variables in α, FV (φ) are free variables in φ, Σ is the set of all variables, and A\B denotes the set of variables being in some set A but not in some other set B. Furthermore, we use v|_A to denote v projected onto the variables in A.

Theorem 1 (Model monitor correctness). Let α* be provably safe, so

Φ→[α*]ψ. Let V_m=BV (α) ∪ FV (ψ). Let v₀, v₁, v₂, v₃ . . . ∈

ⁿ be a sequence of states, with v₀

Φ and that agree on Σ\V_m, i.e., v₀|_{Σ\v m} =v_k|_{Σ\V m} for all k. We define (v, v_i+1)

x_m as x_m evaluated in the state resulting from v by interpreting x⁺ as v_i+1(x) for all x ∈V_m, i.e., v_{x +} ^{v i+1(x)}

x_m. If v_i, v_i+1)

x_m for all i<n then we have v_n

ψ where
X _m≡(Φ|_const→

α

Y _{v m} ⁺)  (3)

and Φ|_const denotes the conditions of Φ that involve only constants that do not change in α, i. e., FV (φ|const) ∩ BV (α)=ø.

The approach shown herein to generate monitor specifications from hybrid system models 102 takes a verified

formula (1) as input 100 and produces a monitor x_m in quantifier-free first-order form as output. The algorithm involves the following steps:

• • 1. A
    
    formula (1) about a model α of the form Φ→[α*]ψ is turned into a specification conjecture (3) of the form Φ|_const→
    
    α
    
    Y_{v m} ⁺. See process 102, resulting in monitor specification 104 in FIG. 3.
  • 2.Theorem proving on the specification conjecture (3) is applied until no further proof rules are applicable and only first-order real arithmetic formulas remain open. See process 106, resulting in monitor conditions 108, containing only first-order logic (FOL) in FIG. 3. Process 106 is also shown in more detail in FIG. 4, and an algorithmic representation of process 106 is shown in Section D herein.
  • 3. The monitor specification x_m is the conjunction of the unprovable first-order real arithmetic formulas from open sub-goals.

Generate the monitor conjecture.

formula (1) is mapped syntactically to a specification conjecture of the form (3). By design, this conjecture will not be provable. But the unprovable branches of a proof attempt will reveal information that, had it been in the premises, would make (3) provable. Through Y_{v m} ⁺, those unprovable conditions collect the relations of the posterior state of model α characterized by x⁺ to the prior state x, i.e., the conditions are a representation of (2) in quantifier-free first-order real arithmetic.

Example 2

The specification conjecture for the water tank model is given below. It is constructed from the model by removing the loop, flipping the modality, and formulating the specification requirement as a property, since we are interested in a relation between two consecutive states

and

(recalled by x⁺, f⁺ and t⁺). Using theorem proving, we analyze the conjecture to reveal the actual monitor specification 104.

$\underset{\varphi \mathrm{lowest}}{\underbrace{\varepsilon >0}}\to \langle f\text{:=}*;?\left(-1\le f\le \frac{m-x}{\varepsilon }\right);t\text{:=}0;\left(x^{\prime }=f,t^{\prime }=1\&x\ge 0\bigwedge t\le \varepsilon \right)\rangle \stackrel{\underset{︷}{{{\rm Y}}_{V_m}^{+}}}{\left(x=x^{*}\bigwedge f=f^{*}\bigwedge t=t^{*}\right)}$

Use theorem proving to analyze the specification conjecture. The proof rules of

are used to analyze the specification conjecture x_m. These proof rules syntactically decompose a hybrid model 100 into easier-to-handle parts, which leads to sequents with first-order real arithmetic formulas towards the leaves of a proof. Using real arithmetic quantifier elimination we close sequents with logical tautologies, which do not need to be checked at runtime since they always evaluate to true for any input. The conjunction of the remaining open sequents is the monitor specification 104, which implies (2).

A complete sequence of proof rules applied to the monitor conjecture of the water tank is described in below in Section B. Most steps are simple when analyzing specification conjectures: sequential composition (

;

), nondeterministic choice (

∪

), deterministic assignment (

:=

) and logical connectives (

r etc.) replace current facts with simpler ones or branch the proof. Challenges arise from handling nondeterministic assignment and differential equations in hybrid programs.

First, consider nondeterministic assignment x :=*. The proof rule for non-deterministic assignment (

*

) results in a new existentially quantified variable. By sequent proof rule ∃r, this existentially quantified variable is instantiated with an arbitrary term θ, which is often a new logical variable that is implicitly existentially quantified. Weakening (Wr) removes facts that are no longer necessary.

$\left(\langle *\rangle \right)\frac{\exists X\langle x\text{:=}X\rangle {\varphi }_1}{\langle x\text{:=}*\rangle \varphi }\left(\exists r\right)\frac{\Gamma ⊢\varphi \left(\theta \right),\exists x\varphi \left(x\right),{\mathrm{<figure-callout\; id="2"\; label="\Delta "\; filenames="US10872187-20201222-D00001.png,US10872187-20201222-D00002.png"\; state="\{\{state\}\}">\Delta </figure-callout>}}_2}{\Gamma ⊢\exists x\varphi \left(x\right),\Delta }\left(\mathrm{Wr}\right)\frac{\Gamma ⊢\Delta }{\Gamma ⊢\varphi ,\Delta }$

¹ X is a new logical variable

² θ is an arbitrary term, often a new (existential) logical variable X.

Optimization 1 (Instantiation Trigger). If the variable is not changed in the remaining α, x_i=x_i ⁺ is Y_{V m} ⁺ and X is not bound in Y_{V m} ⁺, then instantiate the existential quantifier by rule ∃r with the corresponding x_i ⁺ that is part of the specification conjecture (i.e., θ=x_i ⁺), since subsequent proof steps are going to reveal θ=x_i ⁺ anyway.

Otherwise, a new logical variable is introduced which may result in an existential quantifier in the monitor specification if no further constraints can be found later in the proof.

Example 3

The corresponding steps in the water tank proof use

*

for the nondeterministic flow assignment (f:=*) and ∃r to instantiate the resulting existential quantifier ∃F with a new logical variable F (plant is an abbreviation for x′=f, t′=1&0≤x

t≤ε). The proof without and with application of Opt. 1 is given as.

Next, differential equations are handled. Even when the differential equation can be solved, existentially and universally quantified variables remain. Inspect the corresponding proof rule from the

calculus. For differential equations it must be proven that there exists a duration t, such that the differential equation stays within the evolution domain H throughout all intermediate times {tilde over (t)} and the result satisfies φ at the end. At this point there are three options:

• • i. Option 1: instantiate the existential quantifier, if it is known that the duration will be t⁺;
  • ii. Option 2: introduce a new logical variable, which is the generic case that always yields correct results, but may discover monitor specifications that are harder to evaluate;

$\left({\langle }^{\prime }\rangle \right)\frac{\exists T\ge 0{\left(\left(\forall 0\le \stackrel{\_}{t}\le T\langle x\text{:=}y\left(\stackrel{\_}{t}\right)\rangle H\right)\bigwedge \langle x\text{:=}y\left(T\right)\rangle \varphi \right)}_1}{\langle x^{\prime }=\theta \&H\rangle \varphi }$ $\left(\mathrm{QE}\right)\frac{{\mathrm{QE}\left(\varphi \right)}_2}{\varphi }$

¹ T and {tilde over (t)} are fresh logical variables and

x :=y(T)

is the discrete assignment belonging to the solution y of the differential equation with constant symbol x as symbolic initial value

² iff Φ≡QE (Φ), Φ is a first-order real arithmetic formula, QE(Φ) is an equivalent quantifier-free formula computable by [7]

• • iii. Option 3: use quantifier elimination (QE) to obtain an equivalent quantifier-free result (a possible optimization could inspect the size of the resulting formula).

Example 4

In the analysis of the water tank example, the differential equation are solved (see

′

and the substitutions f :=F and t :=0 are applied. In the next step (see ∃r,Wr), the existential quantifier ∃T is instantiated with t⁺ (i.e., T=t⁺ using Option 1 with the last conjunct) and uses weakening right (Wr) to systematically get rid of the existential quantifier that would otherwise still be left around by rule ∃r. Finally, quantifier elimination (QE) is used to reveal an equivalent quantifier-free formula, shown as reference number 108 in FIG. 3.

The analysis of the specification conjecture finishes with collecting the open sequents from the proof to create the monitor specification x_{m ≡} ^def

(open sequent) 104. The collected open sequents may include new logical variables and new (Skolem) function symbols that were introduced for nondeterministic assignments and differential equations when handling existential or universal quantifiers. The invertible quantifier rule i∃ is used to re-introduce existential quantifiers for the new logical variables. Often, the now quantified logical variables are discovered to be equal to one of the post-state variables later in the proof, because those variables did not change in the model after the assignment. If this is the case, proof rule ∃σ can be used to further simplify the monitor specification by substituting the corresponding logical variable x with its equal term θ.

$\left(i\exists \right)\frac{\Gamma ⊢\exists X\left({\Lambda }_i\left({\Phi }_i⊢{\psi }_i\right)\right),{\mathrm{<figure-callout\; id="1"\; label="\Delta "\; filenames="US10872187-20201222-D00001.png,US10872187-20201222-D00002.png"\; state="\{\{state\}\}">\Delta </figure-callout>}}_1}{\Gamma ,{\mathrm{<figure-callout\; id="1"\; label="\Phi "\; filenames="US10872187-20201222-D00001.png,US10872187-20201222-D00002.png"\; state="\{\{state\}\}">\Phi </figure-callout>}}_1⊢{\psi }_2,\Delta \cdots \Gamma ,{\mathrm{<figure-callout\; id="1"\; label="\Phi "\; filenames="US10872187-20201222-D00001.png,US10872187-20201222-D00002.png"\; state="\{\{state\}\}">\Phi </figure-callout>}}_1⊢{\psi }_2,\Delta }\left(\exists o\right)\frac{{\varphi \left(\theta \right)}_2}{\exists x\left(x=\theta \bigwedge \varphi \left(x\right)\right)}$

¹ Among all open branches, free logical variable X only occurs in the branches Γ, Φ, ∛, ψ, Δ

² Logical variable x does not appear in term θ

Example 5

The two open sequents of Examples 3 and 4 use a new logical variable F for the nondeterministic flow assignment f :=*. After further steps in the proof, the assumptions reveal additional information F=f⁺. Thus, the existential quantifier is re-introduced over all the open branches (i∃) and substitute f⁺ for F (∃σ). The sole open sequent of this proof attempt is the monitor specification x_m of the water tank model.

Controller Monitor Synthesis

A controller monitor x_c, shown as reference number 112 in FIG. 3, checks that two consecutive states v and w are reachable with one controller execution α_ctrl, i. e., (

,

) ∈ ρ(α_ctrl) with V_c, =BV (α_ctrl) ∪ FV (ψ). Controller monitors 112 are systematically derived in processes 106 and 110 from formulas 104: φ|_const→

α_ctrl

Y_{V c} ⁺. A controller monitor 112 can be used to initiate controller switching similar to Simplex.

Theorem 2 (Controller monitor correctness). Let α of the canonical form α_ctrl; α_plant Assume |=Φ→[α*]φ has been proven with invariant φ as in (1). Let v

Φ|_const

Φ, as checked by x_m(Theorem 1). Furthermore, let {tilde over (v)} be a post-controller state. If (v, {tilde over (v)})

x_c with x_c≡Φ|_const→

α_ctrl

Y_{V c} ⁺ then we have that (v, {tilde over (v)}) ∈ρ(α_ctrl) and {tilde over (v)}|=φ.

Monitoring in the Presence of Expected Uncertainty and Disturbance

Up to now exact ideal-world models have been considered. However, real-world clocks drift, sensors measure with some uncertainty, and actuators are subject to disturbance. This makes the exact models safe but too conservative, which means that monitors for exact models are likely to fall back to a fail-safe controller unnecessarily often. This section is a discussion of how ModelPlex specifications are found such that the safety property (1) and the monitor specification become more robust to expected uncertainty and disturbance. That way, only unexpected deviations beyond those captured in the normal operational uncertainty and disturbance of α* cause the monitor to initiate fail-safe actions.

In

we can, for example, use nondeterministic assignment from an interval to model sensor uncertainty and piecewise constant actuator disturbance, or differential inequalities for actuator disturbance. Such models include non-determinism about sensed values in the controller model and often need more complex physics models than differential equations with polynomial solutions.

Example 6

Clock drift, sensor uncertainty and actuator disturbance are incorporated into the water tank model to express expected deviation. The measured level x, is within a known sensor uncertainty u of the real level x (i.e. x ∈[x−u, x+u]). Differential inequalities are used to model clock drift and actuator disturbance. The clock, which wakes the controller, is slower than the real time by at most a time drift of c; it can be arbitrarily fast. The water flow disturbance is at most d, but the water tank is allowed to drain arbitrarily fast (even leaks when the pump is on). To illustrate different modeling possibilities, we use additive clock drift and multiplicative actuator disturbance.

$0\le x\le m\bigwedge \varepsilon >0\bigwedge c<1\bigwedge 0\le u\bigwedge 0<d\to \hspace{1em}\left[{\left(x_s\text{:=}*;?\left(x-u\le x_s\le x+u\right);f\text{:=}*;?\left(-1\le f\le \frac{m-x_s-u}{d\varepsilon }\left(1-c\right)\right);t\text{:=}0;\left\{x^{\prime }\le \mathrm{fd},1-c\le t^{\prime }x\ge 0\bigwedge t\le \varepsilon \right\}\right)}^{*}\right]\left(0\le x\le m\right)$

Example6 can be analyzed in the same way as the previous examples, with the crucial exception of the differential inequalities. The proof rule (′) cannot be used to analyze this model, because differential inequalities do not have polynomial solutions. Instead, the DR and DE proof rules of

are used to turn differential inequalities into a differential-algebraic constraint form that lets us proceed with the proof. Rule DE turns a differential inequality x′≤θ into a quantified differential equation ∃{tilde over (d)}(x′={tilde over (d)} & {tilde over (d)}≤θ) with an equivalent differential-algebraic constraint. Rule DR turns a differential-algebraic constraint

into another differential-algebraic constraint

, which implies

, written

→

.

$\left(\mathrm{DR}\right)\frac{𝒟\to \delta \left(𝒟\right){\varphi }_1}{\left(\delta \right)\varphi }\left(\mathrm{DE}\right)\frac{\begin{array}{c}\forall X(\exists \stackrel{\_}{d}\left(X=\stackrel{\_}{d}\bigwedge \stackrel{\_}{d}\le \theta \bigwedge H\right)\to \\ X\le \theta \bigwedge H)\left(\exists \stackrel{\_}{d}\left(x^{\prime }=\stackrel{\_}{d}\&\stackrel{\_}{d}\le \theta \bigwedge H\right)\right){\varphi }_2\end{array}}{\left(x^{\prime }\le \theta \&H\right)\varphi }$

¹ differential refinement: differential-algebraic constraints

,

have the same changed variables

² differential inequality elimination: special case of DR, which rephrases the differential inequalities≤as differential-algebraic constraints (accordingly for other or mixed inequalities systems).

Currently, for finding model monitors, our prototype tool solves differential equations by the proof rule (‘). Thus, it finds model monitor specifications for differential algebraic equations with polynomial solutions and for differential algebraic inequalities, which can be refined into solvable differential algebraic equations as in Example 6. For prediction monitors (discussed in Section below)

techniques are used for finding differential variants and invariants, differential cuts, and differential auxiliaries to handle differential equations and inequalities without polynomial solutions.

Monitoring Compliance Guarantees for Unobservable Intermediate States

With controller monitors, non-compliance of a controller implementation with respect to the modeled controller can be detected right away. With model monitors, non-compliance of the actual system dynamics with respect to the modeled dynamics can be detected when they first occur. In such cases, a fail-safe action is switched to, which is verified using standard techniques, in both non-compliance cases. The crucial question is: can such a method always guarantee safety? The answer is linked to the image computation problem in model checking (i. e., approximation of states reachable from a current state), which is known to be not semi-decidable by numerical evaluation at points; approximation with uniform error is only possible if a bound is known for the continuous derivatives. This implies that additional assumptions are needed about the deviation between the actual and the modeled continuous dynamics to guarantee compliance for unobservable intermediate states. Unbounded deviation from the model between sample points just is unsafe, no matter how hard a controller tries. Hence, worst-case bounds capture how well reality is reflected in the model.

A prediction monitor is derived to check whether a current control decision will be able to keep the system safe for time s even if the actual continuous dynamics deviate from the model. A prediction monitor checks the current state, because all previous states are ensured by a model monitor and subsequent states are then safe by (1).

Definition 2 (ε-bounded plant with disturbance δ). Let α_plant be a model of the form x′=θ& H. An ε-bounded plant with disturbance δ, written α_δplant, is a plant model of the form x_o :=0; (f(θ, δ)≤x′≤g(θ, δ) & H

x_o≤ε) for some f, g with fresh variable ε>0 and assuming x_o′=1. That disturbance δ is constant if x ∈ δ; it is additive if f(θ, δ)=θ−δ and g(θ, δ) =θ+δ.

Theorem 3 (Prediction monitor correctness). Let α* be provably safe, i. e., |=Φ→[α*]ψ has been proved using invariant φ as in (1). Let V_p=BV (α) ∪ FV ([α]φ]). Let v|=Φ|_const

φ, as checked by X_m from Theorem 1. Further assume {tilde over (v)} such that (v, {tilde over (v)}) ∈ρ(α_ctrl), as checked by X_c from Theorem 2. If (v, {tilde over (v)})

X_p with X_p≡(Φ|_const

φ)→

α_ctrl

(Y_{V P} ⁺

[α_δplant]φ), then we have for all ({tilde over (v)}, ω) ∈ρ(α_δplant) that ω|=φ.

Remark 2. By adding a controller execution

α_ctrl

prior to the disturbed plant model, we synthesize prediction monitors that take the actual controller decisions into account. For safety purposes, a monitor definition without controller X_p≡(Φ|_const

φ)→[α_δplant]φ could be used, but doing so results in a conservative monitor, which has to keep the CPS safe without knowledge of the actual controller decision.

Decidability and Computability

One useful characteristic of ModelPlex beyond soundness is that monitor synthesis is computable, which yields a synthesis algorithm, and that the correctness of those synthesized monitors with respect to their specification is decidable. See Theorem 4.

Theorem 4 (Monitor correctness is decidable and monitor synthesis computable). Assume canonical models of the form α≡□α_ctrl; α_plant without nested loops, with solvable differential equations in αplant and disturbed plants α_δplant with constant additive disturbance δ (see Definition. 2). Then, monitor correctness is decidable, i. e., the formulas X_m→

α

Y_V ⁺, X_c→

α_ctrl

Y_V ⁺ γ_V ⁺, and X_p→

α_ctrl

(Y_V ⁺

[α_δplant]φ) are decidable. Also, monitor synthesis is computable, i. e., the functions synth_m:

α

Y_V ⁺→X_m, synth_c:

α_ctrl

Y_V ⁺→X_c and synth_p:

α_ctrl

(Y_V ⁺

[α_δplant]φ)→X_p are computable.

Evaluation

A software prototype was developed and integrated into the modeling tool Sphinx to automate many of the described steps. The prototype generates X_m, X_c, and X_p conjectures from hybrid programs, collects open sequents, and interacts with KeYmaera.

To evaluate the method, we created monitors for prior case studies of non-deterministic hybrid models of autonomous cars, train control systems, and robots, for example, adaptive cruise control, intelligent speed adaptation, the European train control system, and ground robot collision avoidance. Table 2 summarizes the evaluation. For the model, the dimension in terms of the number of function symbols and state variables is listed, as is the size of the safety proof (i. e., number of proof steps and branches).

TABLE 2
Monitor complexity case studies

Model

Monitor

proof size

steps (open seq.)

proof steps

	Case Study	dim.	(branches)	dim.	w/Opt. 1	auto	(branches)	size

Xm	Water tank	5	38 (4)	3	16 (2)	20 (2)	64 (5)	32
	Cruise control [18]	11	969 (124)	7	127 (13)	597 (21)	19514 (1058)	1111
	Speed limit [23]	9	410 (30)	6	487 (32)	5016 (126)	64311 (2294)	19850
Xc	Water tank	5	38 (4)	1	12 (2)	14 (2)	40 (3)	20
	Cruise control [18]	11	969 (124)	7	83 (13)	518 (106)	5840 (676)	84
	Robot [22]	14	3350 (225)	11	94 (10)	1210 (196)	26166 (2854)	121
	ETCS safety [35]	10	193 (10)	13	162 (13)	359 (37)	16770 (869)	153
Xp	Water tank	8	80 (6)	1	135 (4)	N/A	307 (12)	43
http://www.cs.cmu.edu/-smitsch/resource/modelplex_study.zip

For the monitor, the dimension of the monitor conjecture is listed in terms of the number of variables. The number of steps and open sequents are compared when deriving the monitor using manual proof steps to apply Option 1 and fully automated w/o Option 1, and the number of steps in the monitor correctness proof. Finally, the monitor size in terms of arithmetic, comparison, and logical operators in the monitor formula is listed.

Although the number of steps and open sequents differ significantly between manual interaction for Option 1 and fully automated synthesis, the synthesized monitors are logically equivalent. But applying Option 1 usually results in structurally simpler monitors, because the conjunction over a smaller number of open sequents (See Table 2) can still be simplified automatically. The model monitors for cruise control and speed limit control are significantly larger than the other monitors, because their size already prevents automated simplification by Mathematica. As future work, KeYmaera will be adapted to allow user-defined tactics to apply Option 1 automatically.

A. Proofs A.1 Formal Semantics of

ModelPlex bases on a reachability relation semantics instead of trace semantics, since it is easier to handle and suffices for checking at sample points.

The semantics of

, as defined in [27], is a Kripke semantics in which states of the Kripke model are states of the hybrid system. Let

denote the set of real numbers. A state is a map v:V→

; the set of all states is denoted by Sta. We write

|=Φ if formula Φ is true at state v (Def. 4). Likewise,

θ

, denotes the real value of term θ at state

. The semantics of HP α is captured by the state transitions that are possible by running α. For continuous evolutions, the transition relation holds for pairs of states that can be interconnected by a continuous flow respecting the differential equation and invariant region. That is, there is a continuous transition along x^I=θ & H from state

to state

, if there is a solution of the differential equation x^I=θ that starts in state

and ends in

and that always remains within the region H during its evolution.

Definition 3 (Transition semantics of hybrid programs). The transition relation ρ specifies which state

is reachable from a state v by operations of α. It is defined as follows.

• • 1. (ν,ω)∈ρ(x:=θ) iff
    
    z
    
    _ν=
    
    z
    
    _ωf·a·z≠x and
    
    x
    
    _ω=
    
    θ
    
    _ν.
  • 2. (ν,ω)∈ρ(x:=*) iff
    
    z
    
    _ν=
    
    z
    
    _ωf·a·z≠x.
  • 3. (ν,ω)∈ρ(?φ) iff ν=ω and ν
    
    φ.
  • 4. (ν,ω)∈ρ(x′₁=θ₁, . . . ,x′_n=θ_n&H) iff for some r≤0, there is a (flow) function φ:[0,r]→Sta with φ(0)=ν,φ(r)=ω, such that for each time ζ∈[0, r]:
    • (i) The differential equation holds, i.e.,

$\frac{{d\left[x\right]}_w}{\mathrm{dt}}\left(\zeta \right)=||{\theta }_i{||}_{\psi \left(\zeta \right)}$
for each x_i,. (ii) For other variables y∉{x₁, . . . ,x_n, } the value remains constant, i.e.,

y

_φ(ζ)=

y

_φ(0).

• • • (iii) The invariant is always respected, i.e., φ(ζ)
      
      H.
  • 5. ρ(α∪β)=ρ(α)∪ρ(β)
  • 6. ρ(α;β)={(ν,ω):(ν,z)∈ρ(α),(z,ω)∈ρ(β) for a state z}
  • 7. ρ(α*)=∪_n∈Nρ(αⁿ) where αⁱ⁺¹{circumflex over (=)}(α;αⁱ)and α⁰ {circumflex over (=)}?true.

Definition 4 (Interpretation of

formulas). The interpretation |= of a

formula with respect to state ν is defined as follows.

1. ν

θ₁˜θ₂iff[θ_{1 v} ]˜[θ_{2 v} for ˜∈]{=≤, <, ≥, >}

2. ν

φΛΦiff ν

Φ and ν

ψ, accordingly for —, V,→⇄

3. ν

∀x φ iff ω

Φ for all ω that agree with v except for the value of x

4. ν

∃xΦiff ω

Φ for some ω that agrees with ν except for the value of x

5. ν

[α]Φ iff ω

Φ∀ω with (ν, ω) ∈ρ(α)

6. ν

[α]Φ iff w

∃w with (ν, ω)∈ρ(α)

We write |= Φ to denote that Φ is valid, i.e., that v |=Φ∀v.

A.2 Soundness

We recall Lemma 1.

Lemma 1 (Loop prior and posterior state). Let α be a hybrid program and α* be the program that repeats α arbitrarily many times. Assume that all consecutive pairs of states (v_i−1, V_i) ∈ρ(α) of n ∈

⁺ executions, whose valuations are recalled with Y_V ⁱ≡

_x∈_Vx=x^{i and Y} _V ⁱ⁻¹ are plausible w.r.t. the model α, i.e., |=

_1≤i≤n(Y_V ⁱ⁻¹→

α

Y_V ⁱ) with Y_V ⁻=Y_V ⁰ and Y_V ⁺=Y_V ⁿ. Then, the sequence of states originates from an α* execution from Y_V ⁰ to Y_V ⁿ, i.e., |=Y_V ⁻→

α*

Y_V ⁺.

Proof. Follows directly from the transition semantics of α*: ρ(α*) =

where αⁱ⁺¹≅(α; αⁱ) and α⁰ ≅? true.

We recall Theorem 1.

Theorem 1 (Model monitor correctness). Let α* be provably safe, so |=Φ→[α*]ψ. Let V_m=BV (α) ∪ FV (ψ). Let v₀, v₁, v₂, v₃ . . . ∈

ⁿ be a sequence of states, with v₀

Φ and that agree on Σ/V_m, i.e.,

₀|_{Σ/V m} =

_k|_{Σ/V m} for all k. We define (

,

_i+1)

x_m as x_m evaluated in the state resulting from

by interpreting x⁺ as

_i+1(x) for all x ∈ V_m, i.e.,

_{x +} ^{v i+1(x)}

x_m. If

_i,

_i+1)

x_m for all i< n then we have

_n

where
X _m≡(Φ|_const→

α

Y_{V m} ⁺)  (3)

and Φ|_const denotes the conditions of Φ that involve only constants that do not change in α, i.e., FV (Φ|_const) ∩BV(α)=ø.

Proof. By induction over n. If n=0 then (

₀,

₀) ∈ρ(α*)trivially by definition of ρ and

Φ→[α*]ψ implies

₀

ψ. For n>0 assume (

₀,

_n) |ρ(α*) and (

_n,

_n+1)

α

Λ_{x∈V m} x=x⁺. Then there exists μ such that (v_{n +} ^{v n+1(x)} , μ) ∈ρ(α) and the two states agree on all variables except the ones modified by α, i.e., v_{n x +} ^{v n+1(x)} |_Σ/BV(α=μ|_Σ/BB(α). Thus, μ

Y_{V m} ⁺, i.e. μ

_{x∈V m} x=x⁺, which in turn yields μ(x)=μ(x⁺)=(v_{n x +} ^{v n+1(x)} (x⁺)=v_n+1(x) (in other words, μ|V_m=v_n+1|V_m). Since also v_n|_{Σ/V m} we get μ=

_n+1 and (

_n,

_n+1) ∈ρ(α). Hence (

₀,

_n+1) ∈ρ(α*) because by induction hypothesis (

₀,

_n) ∈ρ(α*) and we conclude

_n+1

ψ by assumption

Φ→[α]ψ using

₀

Φ.

We recall Theorem 2.

Theorem 2 (Controller monitor correctness). Let α of the canonical form α_ctrl; α_plant. Assume

Φ→[α*]ψ has been proven with invariant φ as in (1). Let

Φ|_const

ψ, as checked by x_m (Theorem 1). Furthermore, let

be a post-controller state. If (v,

)

X_c with X_c≡Φ|_const→

α_ctrl

Y_{V c} ⁺ = then we have that (v,

) ∈ρ(α_ctrl) and

φ.

Proof. Consider a state

Φ|_const

φ. Assume (v,

)

X_c, i.e.,

X_c. Then there exists μ such that

, μ) ∈ρ(α_ctrl) and the two states agree on all variables except the ones modified by α_ctrl, i.e.,

|_Σ/BV(α_ctrl))=μ|_Σ/BV(α_ctrl). Thus

Y_{V c} ⁺, i.e., μ|=

_{xeV c} x=x⁺, which in turn yields μ(x)=μ(x⁺)=

(x⁺)=

(x) (in other words, μ|V_c=

V_c). Since also μ|_{Σ.V c} =

_{Σ/V c} we get μ=

and (v,

) ∈ρ(α_ctrl). Then we have

φ because by assumption φ→[α_ctrl;α_plant]φ ρ(α_plant) is reflexive as ODE can evolve for time 0.

We recall Theorem 3.

Theorem 3 (Prediction monitor correctness). Let α* be provably safe, i.e.,

Φ→[α*]ψ has been proved using invariant φ as in (1.) Let V_p=BV(α) ∪FV([α]φ). Let

Φ|_const

φ, as checked by X_m from Theorem 1. Further assume

such that (

,

) ∈ρ(α_ctrl), as checked by X_c from Theorem 2. If (

,

)

X_p with X_p≡(Φ|_const

φ) →

α_ctrl

Y_{V p} ⁺

[α_δplant]φ), then we have for all (

,w) ∈ρ(α_δplant) that w

φ.

Proof. Consider a state

such that

Φ|_xonst

φ. Let

be some state such that (

,

) ∈ρ(α_ctrl). Then we have

φ because by assumption φ→[α_ctrl; α_plant]φ and ρ(α_δplant) is reflexive as ODE can evolve for time 0. Furthermore

Φ|_const since

|_Σ/BV(α_ctrl)=

|_Σ/BV(α_ctrl) and FV(Φ|_const) ∩BV(α_ctrl) =ø. Assume (

,

)

X_p, i.e.,

X_p. Then there exists μ such that μ

Y_{V p} ⁺

[α_δplant]φ with (

, μ) ∈ρ(α_ctrl) and the two states agree on all variables except the ones modified by α_ctrl, i.e.,

|_Σ/BV(α_ctrl)=μ|_Σ/BV(α_ctrl). Thus μ(x)=μ(x⁺)=

(x⁺)=

(x). (in other words, μ|V_p=

|V_p). However, from X_p we know that μ

[α_δplant]φ. Thus, by the coincidence lemma,

[α_δplant]φ since FV ([α_δplant]φ) ⊂ V_p and hence we have w

φ for all (

,w) ∈ρ(α_δplant).

Observe that this is also true for all intermediate times ç∈[0, ω(t)] by the transition semantics of differential equations, where ω(t)≤∈ because α_δplant is bounded by ∈.

A.3 Decidability and Computability

From Lemma 1 it follows that online monitoring (i.e., monitoring the last two consecutive states) is permissible. So, ModelPlex turns questions [α*]Φ and

a*

Φ into [α]Φ and

α

Φ, respectively. For decidability, we first consider canonical hybrid programs α of the form α≡α_ctrl; α_plant where α_ctrl and α_plant are free of further nested loops.

We split Theorem 4 (decidability and computability) into Theorem 5 (decidability) and Theorem 6 (computability) and prove them separately. To handle differential inequalities in

formulas of the form [α_δplant]Φ, the subsequent proofs additionally assume the rules for handling differential-algebraic equations in the

calculus.

Theorem 5 (Monitor correctness is decidable). Monitor correctness is decidable for canonical models of the form α≡α_ctrl; α_plant without nested loops, with solvable differential equations in α_plant and disturbed plans α_δplant with constant additive disturbance δ i.e., X_m→

α

Y_V ⁺,X_c→

α_ctrl

Y_V ⁺, and X_p→

α

Y_V ⁺

[α_δplant]Φ) are decidable.

Proof. From relative decidability of

(i.e.,

formulas without free variables) are decidable relative to an oracle for discrete loop invariants/variants and continuous differential invariants/variants. Since neither α_ctrl nor α_plant contain nested loops, we manage without an oracle for loop invariants/variants. Further, since the differential equation systems in α_plant are solvable, we have an effective oracle for differential invariants/variants. Let Cl_∀(Φ) denote the universal closure of

formula Φ (i.e., Cl_∀(Φ)≡∀_z∈FV(Φ)z. Φ). Note that when

F then also

Cl_∀(F) by a standard argument.

Model monitor X_m→

α

Y_V ⁺: Follows from relative decidability of

, because Cl_∀(X_m

α

Y_V ⁺) contains no free variables.

Controller monitor X_c→

α_ctrl

Y_V ⁺: Follows from relative decidability of

, because Cl_∀(X_c→

α_ctrl

Y_V ⁺) contains no free variables.

Prediction monitor X_p

α_ctrl

(Y_V ⁺

[α_δplant]Φ):-Decidability for α_ctrl follows from case X_c→

α_ctrl

Y_V ⁺ (controller monitor) above. It remains to show decidability of X_p→

α_ctrl

[α_δplant]Φ, which by decidability of the controller monitor is (X_p

Y_V ⁺)→[α_plant]Φ. Since the disturbance δ in α_δplant is constant additive and the differential equations in α_plant are solvable, we have the disturbance functions f(θ,δ) and g(θ,δ) applied to the solution as an oracle (By design, the disturbed plant α_δplant also includes a clock x₀, so the oracle additionally includes the trivial differential invariant x₀≥0) for differential invariants (i. e., the differential invariant is a pipe around the solution without disturbance). Specifically, to show (X_p

Y_V ⁺)→[α_δplant]Φ by Def. 2 we have to show (X_pΛY_V ⁺)→[x₀:=0; {θ−δ≤x′≤θ+δ&H Λx₀ ≤ε}]Φ We proceed with only (X_pΛY_V ⁺) →[x₀:=0{x′≤θ+δ&H Λx₀≤ε}]Φ p since the case θ−δ≤x′ follows in a similar manner. By definition of α_δplant we know 0≤x₀, and hence continue with (X_pΛY_V ⁺)→[{x′≤θ+δ&HΛ0≤x₀≤ε}]Φ by differential cut 0≤x₀. Using the differential cut rule, we further supply the oracle sol_x+δx₀, where sol_x denotes the solution of x′=θ in α_plant and δx₀ the solution for the disturbance since δ is constant additive. This leads to two proof obligations:

Prove oracle (X_pΛY_V ⁺)→[x′≤θ+δ&0≤x₀≤ε]x≤sol_x+δx₀, which by rule differential invariant is valid if we can show 0≤x₀≤ε→x′≤sol′_x+(δx₀)′ where the primed variables are replaced with the respective right-hand side of the differential equation system. From Def. 2 we know that x′₀=1 and δ′=0 and since sol_x is the solution of x′=θ in α_plant we further know that sol′_x=θ; hence we have to show 0≤x₀≤ε→θ+δ≤θ+δ, which is trivially true.

Use oracle (X_pΛY_V ⁺)→[x′≤θ+δ&HΛ0≤x₀≤εΛx≤sol_x+δx₀]φ, which by rule differential weaken is valid if we can show
(X _p ΛY _V ⁺)→∀^α(( HΛ0≤x ₀≤εΛx≤sol_x +δx ₀)→Φ)

where ∀^α denotes the universal closure w.r.t. x, i. e., ∀x. But, if X_p is a correct monitor, this is provable by quantifier elimination. Furthermore, we cannot get a better result than differential weaken, because the evolution domain constraint contains the oracle's answer for the differential equation system, which characterizes exactly the reachable set of the differential equation system.

We conclude that the oracle is proven correct and its usage is decidable.

For computability, we start with a theoretical proof on the basis of decidability, before we give a constructive proof, which is more useful in practice.

Theorem 6 (Monitor synthesis is computable). Synthesis of X_m, X_c, and X_p monitors is computable for canonical models of the form α≡α_ctrl; α_plant without nested loops, with solvable differential equations in α_plant and plants α_δplant with constant additive disturbance δ, i. e., synth_m:

α

Y_V ⁺→X_m, synth_c:

α_ctrl

Y_V ⁺→X_c, and synth_p:

α

(Y_V ⁺

[α_plant]Φ)→X_p are computable.

Proof. Follows immediately from Theorem 5 with recursive enumeration of monitors.

We give a constructive proof of Theorem 6. The proof is based on the observation that, except for loop and differential invariants/variants, rule application in the

calculus is deterministic: we know that, relative to an oracle for first-order invariants and variants, the

calculus gives a semidecision-procedure for

formulas with differential equations having first-order definable flows.

Proof. For the sake of a contradiction, suppose that monitor synthesis stopped with some open sequent not being a first-order quantifier-free formula. Then, the open sequent either contains a hybrid program with nondeterministic repetition or a differential equation at top level, or it is not quantifier-free. But this contradicts our assumption that both α_ctrl and α_plant are free from loops and that the differential equations are solvable and disturbance is constant, in which case for

Model monitor synthesis X_m: the solution rule

′

would make progress, because the differential equations in α_plant are solvable; and for

Prediction monitor synthesis X_p: the disturbance functions f(θ,δ) and g(θ,δ) applied to the solution provide differential invariants (see proof of Theorem 5) so that the differential cut rule, the differential invariant rule, and the differential weakening rule would make progress.

In the case of the open sequent not being quantifier-free, the quantifier elimination rule QE would be applicable and turn the formula including quantifiers into an equivalent quantifier-free formula. Hence, the open sequent neither contains nondeterministic repetition, nor a differential equation, nor a quantifier. Thus we conclude that the open sequent is a first-order quantifier-free formula.

B. Water Tank Monitor Specification Conjecture Analysis

Proof 1 shows a complete sequence of proof rules applied to the water tank specification conjecture of Example 2 on page 7, with Φ≡ε>0 and Y⁺≡x=x⁺Λf =f⁺Λt =t⁺.

$\left(\Lambda r\right)\frac{\Gamma ⊢\varphi ,\Gamma ⊢\psi ,\Delta }{\Gamma ⊢\varphi \bigwedge \psi ,\Delta }\left(\mathrm{Wr}\right)\frac{\Gamma ⊢\Delta }{\Gamma ⊢\varphi ,\Delta }\left(\mathrm{QE}\right)\frac{{\mathrm{QE}\left(\varphi \right)}_1}{\varphi }$ $\left(\langle ;\rangle \right)\frac{\langle \alpha \rangle \langle \beta \rangle \varphi }{\langle \alpha ;\beta \rangle \varphi }\left(\langle ?\rangle \right)\frac{H\bigwedge \psi }{\langle ?H\rangle \psi }\left(\langle \text{:=}\rangle \right)\frac{{\varphi }_x^{\theta }}{\langle x\text{:=}\theta \rangle \varphi }\left(\langle *\rangle \right)\frac{\exists X\langle x\text{:=}X\rangle {\varphi }_2}{\langle x\text{:=}*\rangle \varphi }$ $\left({\langle }^{\prime }\rangle \right)\frac{\exists t\ge 0{\left(\left(\forall 0\le \stackrel{\_}{t}\le t\langle x\text{:=}y\left(\stackrel{\_}{t}\right)\rangle H\right)\bigwedge \langle x\text{:=}y*\left(t\right)\rangle \varphi \right)}_3}{\langle x^{\prime }=\theta \&H\rangle \varphi }$ $\left(\exists r\right)\frac{\Gamma ⊢\varphi \left(\theta \right),\exists x\varphi \left(x\right),{\mathrm{<figure-callout\; id="4"\; label="\Delta "\; filenames="US10872187-20201222-D00002.png"\; state="\{\{state\}\}">\Delta </figure-callout>}}_4}{\Gamma ⊢\exists x\varphi \left(x\right),\Delta }$ $\left(i\exists \right)\frac{\Gamma ⊢\exists X\left({\Lambda }_i\left({\Phi }_i⊢{\psi }_i\right)\right),{\Delta }_5}{\Gamma ,{\mathrm{<figure-callout\; id="1"\; label="\Phi "\; filenames="US10872187-20201222-D00001.png,US10872187-20201222-D00002.png"\; state="\{\{state\}\}">\Phi </figure-callout>}}_1⊢{\psi }_1,\mathrm{\Delta \cdots }\Gamma ,{\Phi }_n⊢{\psi }_n,\Delta }\left(\exists \sigma \right)\frac{{\varphi }_x^{\theta }}{\exists x\left(x=\theta \bigwedge \varphi \left(x\right)\right)}$

¹ iff Φ≡QE(Φ), Φ is a first order real arithmetic formula, QE)(Φ) is a quantifier free formula

² X is a new logical variable p ³ t and {tilde over (t)} are fresh logical variables and

x:=y(t)

is the discrete assignment belonging to the solution y of the differential equation with constant symbol x as symbolic initial value

⁴ θ is an arbitrary term, often a new (existential) logistical variable X.

⁵ Among all open branches, free logical variable X only occurs in the branches Γ, Φ, ├ψ, Δ

Proof 1. Analysis of the water tank monitor specification conjecture (plant is an abbreviation for x′=f, t′=1 & x≥0

t≤ε)

Proof 1: Analysis of the water tank monitor specification conjecture (plant is an abbreviation for x′=f, t′=1 & x ≥0

t≤ ε)

B. 1 Monitoring in the Presence of Expected Uncertainty and Disturbance Example 7

We start at the point where we have to handle the differential inequalities. First, we eliminate the differential inequalities by rephrasing them as differential-algebraic constraints in step (DE). Then, we refine by instantiating the existential quantifiers with the worst-case evolution in step (DR). The resulting differential equation has polynomial solutions and, thus, we can use

′

and proceed with the proof as before.

As expected, we get a more permissive monitor specification. One conjunct of the monitor specification is shown in (X_m1) Such a monitor specification says that there exists a real flow F, a real time T, and a real level X_s, such that the measured flow f⁺, the clock t⁺, and the measured level x⁺ can be explained with the model.

C. Monitor Synthesis and Fallback Controller Design C.1 Design-By-Contract Monitoring

Preconditions, postconditions and invariants are crucial conditions in CPS design. Monitors for these conditions can check (i) whether or not it is safe to start a particular controller (i. e., check that the precondition of a controller is satisfied), (ii) whether or not a controller complies with its specification (i. e., check that a controller delivers set values that satisfy its post condition), and (iii) whether or not the system is still within its safety bounds (i. e., check that the loop invariant of α* is satisfied).

Precondition and post condition monitors are useful to decide whether or not it is safe to invoke a controller in the current state, and whether or not to trust a controller output. An invariant monitor of a CPS α* captures the main assumptions that have to be true throughout system execution. When an invariant monitor is unsatisfied, it may no longer be safe to run the CPS; a fail-safe controller can act as a mitigation strategy.

Design-by-contract monitors are useful to monitor specific design decisions, which are explicitly marked in the model. Our approach systematically creates monitors for a complete specification of the behavior of the model.

C.2 Monitor Synthesis

Once we found a model monitor, controller monitor, or prediction monitor specification, we want to turn it into an actual monitor implementation (e. g., in C). The main challenge is to reliably transfer the monitor specification, which is evaluated on

, into executable code that uses floating point representations. We use the interval arithmetic library Apron to represent each real arithmetic value with an interval of a pair of floating point numbers. The interval reliably contains the real.

For certification purposes one still has to argue for the correctness of the actual machine code of the synthesized monitor. This entails that the transformation from the monitor specification as a first-order formula into actual code that evaluates the formula must be formally verified. If the synthesized code is still a high-level language, a certified compiler, can be used to produce machine code. Such a comprehensive proof chain suitable for certification is part of our ongoing research.

C.3 Designing for a Fail-Safe Fallback Controller

When we design a system for a fail-safe fallback controller ctrl_safe, it is important to know within which bounds the fail-safe controller can still keep our CPS safe, and which design limits we want a controller implementation to obey. The invariant of a CPS with the fail-safe fallback controller describes the safety bounds. When we start the fail-safe fallback controller ctrl_safe in a state where its invariant G is satisfied, it will guarantee to keep the CPS in a state that satisfies the safety property ψ.

So, to safely operate an experimental controller ctrl_exp, we want a monitor that informs us when the experimental controller can no longer guarantee the invariant of the fail-safe controller or when it is about to violate the design limits.

A design for a CPS with a fail-safe fallback controller, therefore, involves proving two properties. First, we prove that the fail-safe controller ctrl_safe ensures the safety property ψ as in formula (4) below. This property is only provable if we discover an invariant G for the CPS with the fail-safe controller. Then we use G as the safety condition for generating a prediction monitor.
ø→[(ctrl_safe;plant)*@inv(G)]ψ

With this generic structure in mind, we can design for a fallback controller invoked by a model monitor X_m, controller monitor X_c, or prediction monitor X_p. Upon violation of either X_m, X_c, or X_p by the actual system execution, the set values of a fail-safe controller are used instead.

D. Monitor Synthesis Algorithm

Algorithm 1 lists the ModelPlex specification conjecture analysis algorithm 106, which turns a specification conjecture into an actual monitor, and is shown in flow-chart form in FIG. 4. The algorithm takes a hybrid system model α, a set of variables V that we want to monitor, and an initial condition ø including constraints on the variables not changed in α. (Usually, we want a monitor for all the bound variables of the hybrid system model, i.e., V=BV (α).

Algorithm 1: ModelPlex monitor synthesis

	input:		A hybrid program α, a set of variables V ⊂ BV (α), an initial condition ϕ such
			that |= ϕ → [α*] ψ,
	output:		A monitor Xm such that |= Xm ≡ ϕ|const → α  r+.

	begin	S ← ∅
		γ+ ← x∈vx = x+ with fresh variables xi +	/ / Monitor conjecture
		G ← {├ ϕ|const → α γ+}
1		while G ≠ ∅ do	/ / Analyze monitor conjecture

			foreach g ∈ G do
			G ← G − {g}
			if g is first-order then
			if |≠ g then S ← S ∪ {g}
			else
			{tilde over (g)} ← apply dL proof rule to g
			G ← G ∪ {{tilde over (g)}}

		Xm ← s∈Ss	/ / Collect open sequents

E. Simulation

To illustrate the behavior of the water tank model with a fallback controller, we created two monitors: Monitor X_m validates the complete model (as in the examples throughout this work) and is executed at the beginning of each control cycle (before the controller runs). Monitor X_c validates only the controller of the model α (compares prior and post state of f:=*; ?

$-1\le f\le \frac{m-x}{\varepsilon })$
the controiier but before control actions are issued. Thus, monitor X_c resembles conventional runtime verification approaches, which do not check CPS behavior for compliance with the complete hybrid model. This way, we detect unexpected deviations from the model at the beginning of each control cycle, while we detect unsafe control actions immediately before they are taken. With only monitor X_m in place we would require an additional control cycle to detect unsafe control action, whereas with only monitor X_c in place we would miss deviations from the model. Note that we could run monitor X_m in place of X_c to achieve the same effect. But monitor X_m implements a more complicated formula, which is unnecessary when only the controller output should be validated.

FIG. 2 shows a plot of the variable traces of one simulation run. In the simulation, we ran the pump controller every 2 s (ε=2 s, indicated by the grid for the abscissa and the marks on sensor and actuator plots). The controller was set to pump with

$\frac{5\left(m-x_0\right)}{\varepsilon }=\frac{5}{2}$
for the first three controller cycles, which is unsafe on the third controller cycle. Monitor B immediately detects this violation at t=4, because on the third controller cycle setting

$f=\frac{5}{2}$
violates

$f\le \frac{m-x_1}{\varepsilon }.$
The fail-safe action at t=4 drains the tank and, after that, normal operation continues until t=12. Unexpected disturbance

$x^{\prime }=f+\frac{1}{20}$
occurs throughout t=[12,14], which is detected by monitor X_m. Note, that such a deviation would remain undetected with conventional approaches (monitor X_c is completely unaware of the deviation). In this simulation run, the disturbance is small enough to let the fail-safe action at t=14 keep the water tank in a safe state.

Claims (20)

We claim:

1. A computer-implemented method for synthesizing a monitor for a cyber-physical system (CPS) from a verified model of the CPS, the verified model expressed as a set of mathematical formulas, comprising the steps of:

deriving a monitor specification from the verified model, the monitor specification comprising a set of specification conjectures, each specification conjecture derived from one of the set of mathematical formulas by verified transformation such that each specification conjecture is satisfied over states of the CPS only if the states are compatible with behavior of the verified model;

applying a theorem prover to each specification conjecture to derive one or more monitor conditions expressed as first-order logic by performing logical transformations that successively simplify formulas into structurally simpler formulas expressing effects of the verified model behavior;

synthesizing the monitor by syntactic transformation or compilation of the monitor conditions to executable code that verifies the monitor conditions; and

verifying the behavior of the CPS by executing the monitor to determine that the CPS meets the set of monitor conditions indicating compliance with the verified model;

wherein the CPS includes a controller running control software, the controller reading sensors and controlling actuators, and wherein the monitor verifies the actions of the controller and resulting changes in a state of the CPS.

2. The method of claim 1 further comprising the step of executing the monitor in conjunction with the control software, to guarantee compliance of the CPS with the verified model.

3. The method of claim 1 where the monitor consists of a model monitor, a controller monitor and a prediction monitor.

4. The method of claim 3 wherein the model monitor compares a current state of the CPS with a previous state of he CPS and determines if any differences between the current state and the previous state are in compliance with the verified model.

5. The method of claim 3 wherein the controller monitor monitors the controller to check compliance with the verified model.

6. The method of claim 3 wherein the prediction monitor allows deviations of the cyber-physical system from the verified model to account for real-world imperfections.

7. The method of claim 4 wherein the model monitor, the controller monitor and the prediction monitor read the sensors to determine the current state of the CPS.

8. The method of claim 7 wherein the model monitor runs prior to the control software and determines if physical behavior which occurred between the previous state and the current state is in compliance with the verified model.

9. The method of claim 8 wherein the model monitor will raise an error if any differences between the current state and the previous state are not in compliance with the verified model.

10. The method of claim 5 wherein the controller monitor runs after the control software and determines if physical actions to be taken by the controller will result in the CPS being in compliance with the verified model.

11. The method of claim 10 wherein the controller monitor will raise an error if the physical actions will result in the CPS not being in compliance with the verified model.

12. The method of claim 6 wherein the prediction monitor runs after the control software and determines if physical actions to be taken by the controller will result in the CPS being in compliance with the verified model in the presence of bounded deviations between the CPS and the verified model.

13. The method of claim 12 wherein the prediction monitor will raise an error if the physical actions to be taken by the controller will result in the CPS not being in compliance with the verified model.

14. The method of claim 1 wherein the verified model comprises a set of formulas of the form:

Φ→[α*]ψ with invariant ψ→[α]ψ s.t. Φ→φ and φ→ψ  (1)

where:

Φ is a precondition for the run;

α* describes repeated runs of the hybrid system;

α is the hybrid system model;

ψ is a postcondition for the run; and

φ is a loop invariant.

15. The method of claim 8 wherein compliance with the model monitor guarantees that properties of the verified model are present in the CPS before the control software is executed.

16. The method of claim 10 wherein compliance with the controller monitor guarantees that properties of the verified model are present in the CPS after the control software is executed.

17. The method of claim 12 wherein compliance with the prediction monitor guarantees that properties of the verified model will still be present in future states of the CPS.

18. The method of claim 1 wherein the method is automated in accordance with an algorithm.

19. The method of claim 1 wherein the mathematical formulas are differential dynamic logic formulas.

20. The method of claim 3, wherein the prediction monitor predicts a next state of the CPS based on the actions to be taken by the controller and initiates failsafe actions if the next state of the CPS deviates from the verified model.

Images