The Kalman.prm file controls the Kalman filter. It has two parts, in this order:
•Options: lines that begin with a colon, such as :DEBUG or :KFVERSION=2. Lines beginning :COM are comments.
•Covariance lines, which give the driving (process) noise and the initial covariance for the model's stocks, and optionally the noise on the flows between them.
Options must come before the first covariance line. An option that appears after it is ignored. Comment lines and blank lines can go anywhere.
The quickest way to start a file for a model is to run the filter once with :TEMPLATE. Vensim writes a complete Kalman.prm for that model, listing every stock the filter carries, which you can then edit. See Generating a starting file, below.
The filter's matrices
Several options refer to the matrices the filter works with. Each step, the filter predicts the stocks forward with the model, then corrects them toward the data. The matrices describe that process:
•F, the state transition matrix: how each stock at the next step depends on every stock now. Vensim computes it by linearizing the model at each step, so it follows the model as its behavior changes.
•H, the measurement matrix: how each data series in the payoff depends on the stocks, also from linearizing the model. It is how a correction to a measured variable is shared out among the stocks that drive it.
•Q, the driving noise covariance: how much uncertainty enters each stock per step, and how that uncertainty is shared between stocks. This is what the covariance lines in this file set.
•R, the measurement noise covariance: how uncertain each data series is. It comes from the weights in the payoff definition, not from this file.
•The residual weighting matrix: weights on the residuals when :LEAStsq=1 replaces maximum likelihood with least squares. It is not used otherwise.
The option names follow a pattern. CN means the matrix is held constant rather than recomputed, DI that it is treated as diagonal, and ID that it is replaced by the identity. The last letters name the matrix: HTil, QTil, RTil and FTil for H, Q, R and F, and REswm for the residual weighting matrix. All of these simplify the filter and are rarely appropriate; the defaults are right for nearly every model.
Options
Options are written :KEYWORD=value. Case does not matter, and only the first four letters of a keyword are read, so :CNHT, :CNHTil and :cnhtil are the same option. :DEBUG, :TEMPLATE and :JOSEPH may be written bare to switch them on. Defaults are shown, except for :START, which has none. Most models need only the first few lines.
:COM Units of driving variance: 1 = per TIME STEP (legacy), 2 = per unit time
:KFVERSION=2
:COM Write <run>_kalman.err with the filter's matrices and diagnostics
:DEBUG=0
:COM Write <run>_template.prm, a complete covariance file for this model
:TEMPLATE=0
:COM Reduce the weight of observations more than k standard deviations out; 0 = off
:HUBER=0
:COM Update the state covariance in Joseph form; the default from KFVERSION 2 on
:JOSEPH=1
:COM Standard deviations used by the whiteness test (floating point > 0)
:NSDWt=4
:COM Maximum lag for the whiteness autocorrelations (small integer)
:LAGMax=4
:COM Filter reaches steady state at sample NSS (integer >= 1)
:STEAdy=0
:NSS=1
:COM Least squares in place of maximum likelihood (deprecated)
:LEAStsq=0
:COM H, the measurement matrix, is computed once and not updated
:CNHTil=0
:COM Q, the driving noise covariance, is set at the start and not updated
:CNQTil=0
:COM R, the measurement noise covariance, is held constant (deprecated)
:CNRTil=0
:COM The residual weighting matrix is held constant (least squares only; deprecated)
:CNREswm=0
:COM Q is diagonal: covariances between stocks are ignored
:DIQTil=0
:COM R is diagonal: covariances between measurements are ignored
:DIRTil=0
:COM The residual weighting matrix is diagonal (least squares only; deprecated)
:DIREswm=0
:COM F, the state transition matrix, is the identity: dynamics are ignored
:IDFTil=0
:COM H is the identity: each measurement is the stock in the same position
:IDHTil=0
:COM R is the identity: every measurement has noise variance 1
:IDRTil=0
:COM The residual weighting matrix is the identity (least squares only; deprecated)
:IDREswm=0
:COM No measurement update before this time; omit to filter from the first sample
:START=160
:KFVERSION sets what a driving variance means, and is described below. New files should use 2.
:DEBUG writes <run>_kalman.err. It begins with a numbered list of the states and measurements, which label the rows and columns of every matrix in the file, followed by the initial matrices, the measurement noise at the first sample, and then per-sample details and reports on the conditioning of the covariance.
:HUBER=k makes the filter robust to occasional wild observations. When an observation's residual is more than k standard deviations out, its variance for that update is multiplied by the distance divided by k, so it pulls the estimate less. The point is not discarded and still counts in the payoff. A value around 3 is typical.
:JOSEPH controls how the state covariance is updated when data arrives. It is on by default from :KFVERSION 2 on, and off before that so no older file changes its results; :JOSEPH=0 returns to the older update at any version. That older update combines the predicted covariance with the information in the measurements by inverting both, which is costly when there are many stocks and loses precision when the covariance is poorly conditioned. The Joseph form updates the covariance directly from the Kalman gain, with no inversion of a matrix the size of the state, and keeps it positive semi-definite. It also takes :HUBER downweighting into account, which the older update does not. Without :HUBER the two give the same result apart from rounding. :JOSEPH is ignored with :LEAStsq, and after the steady-state sample when :STEAdy is set.
:CNRTil is deprecated. Holding the measurement noise constant means it is never computed from the payoff weights, so the filter runs with a measurement noise of 1 for every data series, whatever weights the payoff specifies. The weight cannot simply be used as the variance instead: depending on the payoff element it is a variance, a standard deviation or the reciprocal of a standard deviation, and a weight given as a model variable has no single value to hold. Leave :CNRTil at 0. Vensim issues a warning if it is set.
:IDRTil ignores the payoff weights in the same way, giving every data series a measurement noise variance of 1.
:LEAStsq is deprecated, and so are the three residual weighting options, which only it uses. There is no way to supply a residual weighting matrix, so least squares works only with :IDREswm=1, which weights every residual equally; the other two have no values to work with. Maximum likelihood, the default, uses the measurement noise from the payoff weights and is the better choice. Vensim issues a warning if :LEAStsq is set.
:IDHTil is meaningful only when the data series correspond one for one, in order, to the stocks. :IDFTil discards the model's dynamics when propagating uncertainty, and is a diagnostic rather than a modelling choice.
:STEAdy=1 treats the filter as having reached steady state after sample :NSS. From then on the gain and covariance are no longer updated; the stocks are still corrected with the last gain.
:CNQTil holds the driving noise at its initial values. Any variance that is a model variable, scales with a stock or comes from a flow is then frozen at its value at the start of the run.
:DIQTil discards all covariances between stocks, which removes exactly what a flow line supplies. Vensim reports an error if it is combined with flow lines.
Covariance lines
A covariance line names a stock, then its driving variance, then its initial variance, separated by slashes. A line naming two stocks joined by | gives the covariance between them:
stock/driving variance/initial variance
stockA|stockB/driving covariance/initial covariance
Each value may be a number or the name of a model variable. A variable is re-read every time step, so the noise can change during the run. The initial variance may be left off.
Elk/0.34/1000
Wolves/Wolf driving variance/Wolf initial variance
Elk|Wolves/Elk wolf covariance
Names are checked. A name that is not a stock, or a value that is not a variable in the model, stops the run with an error rather than being skipped. This includes the internal levels of macros such as SMOOTH and DELAY3, which can be named as Vensim writes them, for example #Measured Infections>Delay3>LV1#.
Operators
A value may begin with an operator that changes how it is read. With no operator the value is the variance itself.
stock/*x/... x is a standard deviation as a fraction of the stock: (stock*x)^2
stock/~x/... x is a variance as a fraction of the stock: |stock*x|
stock/+flow/... the variance is |flow|, the Poisson noise of that flow
stock/>x/... a floor: the variance is raised to x if it would otherwise be less
stock//init no variance of its own: flow lines supply it
x may be a number or a model variable, so stock/*Stock noise fraction/*0.1 is valid.
* and ~ let the noise follow a stock whose size changes over the run, which a fixed number cannot. Both have a small floor so that a stock sitting at zero does not end up with zero variance, which the filter cannot handle. They scale a single stock, so they cannot be used on a stockA|stockB line.
A relative variance goes to nothing as the stock empties, which is often exactly when the filter most needs some uncertainty. A stock that can be empty for part of the run, such as a count of infected people before an outbreak, usually wants an absolute floor as well, written with >. A floor of 1 on a stock counting people is the Poisson variance of one person.
+ must be followed directly by a name. + followed by a digit is read as a signed number. + cannot be used for an initial variance, and neither can >.
Flow lines
A flow between two stocks takes from one and adds to the other, so any uncertainty in the flow moves the two stocks in opposite directions. A flow line says so:
Susceptible|Infected/+Infection Rate
This adds the flow's variance to both stocks and subtracts it from the covariance between them. The filter can then never push the total of the two stocks away from its value, which matters whenever a model conserves something - a population, an energy balance - and uses that total elsewhere, for example as a divisor. Without flow lines, independent corrections to each stock let the total drift.
The order of the two stocks does not matter. Give one line per flow; several flows between the same stocks accumulate. A flow into or out of the model, with only one stock, adds to that stock alone:
Infected/+Patient Zero Arrival
A flow line between two stocks has no initial covariance field. When a stock's noise comes only from such flows, give it a line with an empty variance to set its initial variance:
Recovered//1000
How lines combine
When several lines name the same stock, three rules apply in this order:
•Variance lines replace one another, and the later line in the file wins. This lets a line for a whole subscript range set a rule and a later line override one element.
•Flow lines add to whatever the variance lines left.
•Floors, written with >, apply last. The result is the larger of the total and the floor.
Infected[Region]/*0.01/*0.01
Infected[Texas]/*0.05/*0.05
Susceptible[Region]|Infected[Region]/+Infection Rate[Region]
Infected[Region]/>1
Here every region gets 1% noise except Texas at 5%, infections add their Poisson noise, and no region's variance falls below 1. A floor line can also carry the initial variance, so the last two ideas can share a line: Infected[Region]/>1/Infected initial variance[Region].
A stock that no line names gets a default. Under :KFVERSION=2 its driving noise and its initial variance both have a standard deviation of 0.1% of the stock, with the same small floor as *. Under version 1 it gets no driving noise and an initial variance of 1.
Subscript ranges
A line may use a subscript range in place of an element, and applies to every element:
Vaccinated[Region]/Vax driving variance[Region]/Vax initial variance[Region]
Susceptible[Region]|Vaccinated[Region]/+Vaccination Rate[Region]
As in .voc and .vpd files, every name on the line must either use the same ranges in the same order as the first name, or use none - a single element, an unsubscripted variable or a number. So stock[r]/var[r]/init[r] and stock[r]/v/3 are valid, while stock[a,b]/var[b] and stock[a,b]/var[a] are errors. Write stock[a,b]/var[a,b] for a variance that does not vary over b.
Driving variance units: KFVERSION
Without a :KFVERSION line, or with :KFVERSION=1, a driving variance is the variance added each TIME STEP. The filter then depends on the integration step: halving TIME STEP doubles the noise accumulated per unit of time. Files written for earlier versions of Vensim work this way and continue to.
With :KFVERSION=2, a driving variance is the variance added per unit of time, like every other rate in a model, and Vensim multiplies it by TIME STEP. Results no longer depend on TIME STEP, and a flow line with no multiplier gives the correct counting noise for that flow. Initial variances are the same under both versions.
Version 2 also changes what unnamed stocks get, as described under How lines combine. Converting a version 1 file by hand means dividing each driving variance by TIME STEP; the template does this for you.
Version 2 additionally corrects how the covariance is grown between data points. Version 1 advanced it by a single TIME STEP per measurement interval, however far apart the measurements were, so a model with data every year and a TIME STEP of 0.125 grew the covariance an eighth as far as it should have. The filter was then too confident of the state, it gave each new observation too much weight, and parameter estimates came out looking more precise than the data support. Version 2 grows the covariance once per TIME STEP. A model with data at every TIME STEP is unaffected, which is why this went unnoticed for a long time. Version 1 files keep the old behaviour so that existing calibrations reproduce; to get the correction, convert the file with :TEMPLATE.
Starting the filter later: START
:START=t holds the filter off until time t. The model simulates normally over that period, but the covariance is held at its initial value rather than grown, and the model is not linearized. The initial covariance is therefore the uncertainty at the moment filtering begins.
Filtering begins at the first data point, or at t if that is later. Under :KFVERSION=2 a t earlier than the first data point moves the beginning the other way: the covariance starts growing at t, while the corrections still wait for data. Use that when the state is genuinely uncertain before the measurements start and you want the filter to say so. :START=0 with an INITIAL TIME of 0 grows the covariance over the whole run. Under version 1 an early t has no effect, because the covariance is not grown before the first data point either way.
Use it when early data cannot inform the state. Reported counts of zero before an outbreak, while the model is already producing cases, are not evidence that the state is zero. A filter given that period spends it absorbing a systematic mismatch into whichever stock is least constrained, and can fail much later as a result. Removing the early data has a similar effect but also changes what the payoff is computed over.
Relinearizing less often: KALMAN LINEAR INTERVAL
The filter needs a linear approximation of the model, the state transition matrix F, to grow the covariance between data points. Computing it perturbs every stock, and by default it is recomputed every TIME STEP once data has started. That is usually most of the cost of a filtered simulation.
Defining a model constant named KALMAN LINEAR INTERVAL with a value greater than 0 recomputes F only at that interval, holding it in between. With data at every unit of time and a TIME STEP of 0.125, an interval of 1 recomputes F once per data point, just after the filter has corrected the state, for about an eighth of the cost. This is an approximation: the held F stops following the model as the state moves, so compare results against a run without it before relying on a long interval. When data is sparse, an interval shorter than the spacing of the data keeps F current between data points.
KALMAN LINEAR INTERVAL is a model variable, not a Kalman.prm option. Leave it out, or set it to 0, to recompute F every TIME STEP. It is ignored when no Kalman filter is running.
Generating a starting file: TEMPLATE
Running the filter with :TEMPLATE writes <run>_template.prm, a complete Kalman.prm for the model in the version 2 form. Rename it and use it in place of the original. It contains:
•Every option, with its value in effect.
•Your own covariance lines, each followed by a comment giving the values it produced at the start of the run. If the file you ran was version 1, its numeric driving variances are divided by TIME STEP so that the filter behaves as before; a variance given as a model variable is flagged with a :COM !! note, since its equation needs the same change by hand.
•A line for every stock the file did not name, including macro internals, giving the version 2 default. A subscripted variable gets one line for its whole range unless you have named one of its elements. These lines change a version 1 filter, which gave such stocks no noise at all.
•The flow structure of the model, as commented flow lines ready to use. Where a stock's rate could not be broken into flows, the variables involved are listed so you can write the line yourself.
Uncommenting a flow line adds to the noise the stock already has. To let the flows supply all of it, empty the variance field of the stock's own line, stock//init, rather than deleting the line, which would also discard its initial variance.
Measurement noise
Measurement noise does not appear in this file. It comes from the weights in the payoff definition; see Extended payoffs for how each payoff type's weight is interpreted. Measurement standard deviations should reflect the real error in the data. For counts, that is at least the Poisson standard deviation, the square root of the count. A sigma set generously for comfort makes large, persistent residuals look normal and hides a model that does not fit.
Counts near zero
The filter assumes the measurement error is Gaussian. A count is not, and at small counts the difference matters: the distribution is skewed, it is discrete, and a large share of the observations are zero. Giving the variance correctly does not fix this, because it is the shape that is wrong, not the spread.
What this costs is confidence bounds that are too narrow rather than estimates that are wrong. In a study of a predator-prey model calibrated against 60 synthetic datasets, the parameter estimates were unbiased and their intervals were correctly calibrated on the datasets where the populations stayed well above zero. On the datasets where the prey population crashed to a handful of animals, about twice as many true values fell outside the 90% interval as should have. The residuals in that regime had the right variance but were visibly skewed, with a third of them zero.
Two things that seem like they should help, and do not. Putting a floor under the state variance makes the intervals no better and the narrower ones worse. Nor does smoothing the model so the population cannot quite reach zero: the estimates barely move.
What does help is comparing a transformed count instead of the count itself, so that what the filter sees is closer to Gaussian. Compare SQRT(data + 3/8) against SQRT(model + 3/8) with a measurement variance of 1/4, in place of comparing the counts directly. This is the Anscombe transform: it makes a Poisson count close to Gaussian with a constant variance even when the mean is small, and the filter handles the square root like any other nonlinearity. Define the transformed quantities as auxiliaries in the model and name those in the payoff.
This corrects the measurement side only. A model whose dynamics are strongly nonlinear where the counts are small - a population close to extinction, for instance - is still being approximated, and the filter can only report what a linear approximation around its current estimate can express.
Errors
Mistakes in the covariance lines stop the run with a message giving the line, rather than being skipped. They include:
•a name that is not a stock, or a value that is not a model variable
•a line that does not start with a stock name, a colon or nothing
•*, ~ or > on a stockA|stockB line, and + or > on an initial variance
•an empty variance on a stock that no flow line or floor supplies
•subscript ranges that do not match the first name on the line
•flow lines combined with :DIQTil
