The payoff report includes a variety of metrics for evaluating the outcome of an optimization, including the component contributions of each payoff element to the total payoff, and the sources of exogenous data (if any).
Metadata & Contributions
•Type - payoff element type, e.g., *P or *CR
•Component - the element specification as model_var|data_var/scale_or_weight
•Contribution - the element's contribution to the total payoff
•Percent - the element's percentage contribution to the total payoff; may be NA if the total is 0 or contributions have mixed sign
•Values - the element's contribution due to model values (for policy elements) or model-data comparisons (calibration elements)
•Params - the element's contribution from scale parameter terms in the likelihood (some calibration distributions only)
•Used - number of points used
•Skipped - number of points skipped, either due to exceeding the model's time range, or invalid comparisons like LN(0)
•Source - data variable source identifier
Goodness of Fit
•RSquare - coefficient of determination, R2 = SSE/SST where SSE = sum of squared errors and SST = total sum of squares. Note that, while R2 is typically bounded between 0 and 1 in typical regression, bias and other factors make it possible for SSE > SST in a nonlinear model, so R2 may be negative.
•DW - Durbin Watson test statistic for detection of autocorrelation of residuals
•Autocor1 - residual autocorrelations at 1 to 4 steps
•Autocor2
•Autocor3
•Autocor4
•(TimeGap) - time gap for residual autocorrelation steps
Theil Statistics
The Theil statistics decompose model-data variance into three terms, from difference in means, difference in variance, and difference in covariance. See:
Sterman, J.D., (1983) Appropriate Summary Statistics for Evaluating the Historical Fit of System Dynamics Models. Dynamica, 10 (Winter), 51-66. https://systemdynamics.org/wp-content/uploads/assets/proceedings/1983/sterm203.pdf
Vensim reports:
•RMSE - the total root mean squared error between model and data
•Um - the fraction of MSE due to difference in means
•Uv - the fraction of MSE due to difference in variance
•Uc - the fraction of MSE due to point-to-point covariance
Um + Uv + Uc = 1
All values will be reported as 0 if there are no data points.
If there are fewer than 2 data points, Uv and Uc will be 0, with Um = 1 and MSE reflecting the squared error of the single data point.
Absolute Errors
•MAE - mean absolute error, i.e. the sum over the valid model & data points of ABS(model-data) divided by the count of comparisons.
•MAPE - mean absolute percentage error, the sum of ABS((data-model)/data) multiplied by 100 to yield a percentage. If any data value is 0 this statistic is undefined and NA is reported.
•MAEoM - mean absolute error over mean, the MAE divided by the mean of the data, multiplied by 100 to yield a percentage. If the data mean is 0, NA is reported.
If there are no data points, 0s are reported.
