Format 1: long view

Format 2: vertical view

Format 3: horizontal view

Depending on the location of your input data, you can use create_vintages_from_xlsx() or create_vintages_from_csv(), or the more generic function create_vintages() to create the vintages (see section vintages & revisions) later on. If you plan to use the generic function, you’ll first need to put your input data in a data.frame in R as in the example below.

# Long format
long_view <- data.frame(
    rev_date = rep(x = c("2022-07-31", "2022-08-31", "2022-09-30", "2022-10-31",
                         "2022-11-30", "2022-12-31", "2023-01-31", "2023-02-28"),
                   each = 4L),
    time_period = rep(x = c("2022Q1", "2022Q2", "2022Q3", "2022Q4"), times = 8L),
    obs_values = c(
        .8, .2, NA, NA, .8, .1, NA, NA,
        .7, .1, NA, NA, .7, .2, .5, NA,
        .7, .2, .5, NA, .7, .3, .7, NA,
        .7, .2, .7, .4, .7, .3, .7, .3
    )
)
print(long_view)

# Horizontal format
horizontal_view <- matrix(data = c(.8, .8, .7, .7, .7, .7, .7, .7, .2, .1,
                                   .1, .2, .2, .3, .2, .3, NA, NA, NA, .5, .5, .7, .7,
                                   .7, NA, NA, NA, NA, NA, NA, .4, .3),
                          ncol = 4)
colnames(horizontal_view) <- c("2022Q1", "2022Q2", "2022Q3", "2022Q4")
rownames(horizontal_view) <- c("2022-07-31", "2022-08-31", "2022-09-30", "2022-10-31",
                               "2022-11-30", "2022-12-31", "2023-01-31", "2023-02-28")
print(horizontal_view)

# Vertical format
vertical_view <- matrix(data = c(.8, .2, NA, NA, .8, .1, NA, NA, .7, .1, NA,
                                 NA, .7, .2, .5, NA, .7, .2, .5, NA, .7, .3, .7, NA,
                                 .7, .2, .7, .4, .7, .3, .7, .3),
                        nrow = 4)
rownames(vertical_view) <- c("2022Q1", "2022Q2", "2022Q3", "2022Q4")
colnames(vertical_view) <- c("2022-07-31", "2022-08-31", "2022-09-30", "2022-10-31",
                             "2022-11-30", "2022-12-31", "2023-01-31", "2023-02-28")
print(vertical_view)

Vintages & revisions

Once your input data are in the right format, you must first create an object of the class rjd3rev_vintages before you can run your revision analysis. The function create_vintages() (or, alternatively, create_vintages_from_xlsx() or create_vintages_from_csv()) will create this object from the input data and display the vintages considering three different data structures or views: vertical, horizontal and diagonal.

1. The vertical view shows the observed values at each time period by the different vintages. This approach is robust to changes of base year and data redefinition and could for example be used to analyse revisions resulting from a benchmark revision. A drawback of this approach is that for comparing the same historical series for different vintages, we need to look at the smallest common number of observations and consequently the number of observations is in some circumstances very small. Moreover, it is often the case that most of the revision is about the last few points of the series so that the number of observations is too small to test anything.

2. The horizontal view shows the observed values of the different vintages by the period. A quick analysis can be performed by rows in order to see how for the same data point (e.g. 2023Q1), figures are first estimated, then forecasted and finally revised. The main findings are usually obvious: in most cases the variance decreases, namely data converge towards the ‘true value’. Horizontal tables are just a transpose of vertical tables and are not used in the tests in ‘revision_analysis’.

3. The diagonal view shows subsequent releases of a given time period, without regard for the date of publication. The advantage of the diagonal approach is that it gives a way to analyse the trade between the timing of the release and the accuracy of the published figures. It is particularly informative when regular estimation intervals exist for the data under study (as it is the case for most of the official statistics). However, this approach requires to be particularly vigilant in case there is a change in base year or data redefinition.

Note that in the argument of the function create_vintages(), the argument vintage_selection allows you to limit the range of revision dates to consider if needed. See ?create_vintages for more details.

Revisions are easily calculated from vintages by choosing the gap to consider. The function get_revisions() will display the revisions according to each view. It is just an informative function that does not need to be run prior the revision analysis.

Revision analysis & reporting

The function revision_analysis() is the main function of the package. It provides some descriptive statistics and performs a battery of parametric tests which enable the users to detect potential bias (both mean and regression bias) and sources of inefficiency in preliminary estimates. We would conclude to inefficiency in the preliminary estimates when revisions are predictable in some way. Parametric tests are divided into 5 categories: relevancy (check whether preliminary estimates are even worth it), bias, efficiency, orthogonality (correlation at higher lags), and signalVSnoise. This function is robust. If for some reasons, a test fails to process, it is just skipped and a warning is sent to users with the possible cause of failure. The other tests are then performed as usual.

For some of the parametric tests, prior transformation of the vintage data may be important to avoid misleading results. By default, the decision to differentiate the vintage data is performed automatically based on unit root and co-integration tests. More specifically, we focus on the augmented Dickey-Fuller (ADF) test to test the presence of unit root and, for cointegration, we proceed to an ADF test on the residuals of an OLS regression between the two vintages. The results of those tests are also made available in the output of the function (section ‘varbased’). By contrast, the choice of log-transformation is left to the discretion of the users based on their knowledge of the series and the diagnostics of the various tests. By default, no log-transformation is considered.

As part of the arguments of the revision_analysis() function, you can choose the view and the gap to consider, restrict the number of releases under investigation when diagonal view is selected and/or change the default setting about the prior transformation of the data (including the possibility to add a prior log-transformation of your data).

The function render_report() applied on the output of revision_analysis() will generate an enhanced HTML or PDF report including both a formatted summary of the results and full explanations about each test (which are also included in the vignette below). The formatted summary of the results display the p-value of each test (except for Theil’s tests where the value of the statistics is provided) and their interpretation. An appreciation ‘good’, ‘uncertain’, ‘bad’ and ‘severe’ is indeed associated to each test following the usual statistical interpretation of p-values and the orientation of the tests. This allows a quick visual interpretation of the results and is similar to what is done in the GUI of JDemetra+.

In addition to the function revision_analysis(), the user can also perform tests individually if they want to. To list all functions available in the package (and therefore finding the different functions corresponding to the individual tests), you can do

ls("package:rjd3revisions")

Use help(‘name of the functions’) or ?‘name of the functions’ for more information and examples over the various functions.

Theil’s Inequality Coefficient

In the context of revision analysis, Theil’s inequality coefficient, also known as Theil’s U, provides a measure of the accuracy of a set of preliminary estimates (P) compared to a latter version (L). There exists a few definitions of Theil’s statistics leading to different interpretation of the results. In this package, two definitions are considered. The first statistic, U1, is given by $$ U_1=\frac{\sqrt{\frac{1}{n}\sum^n_{t=1}(L_t-P_t)^2}}{\sqrt{\frac{1}{n}\sum^n_{t=1}L_t^2}+\sqrt{\frac{1}{n}\sum^n_{t=1}P_t^2}} \\ \\ $$ U1 is bounded between 0 and 1. The closer the value of U1 is to zero, the better the forecast method. However, this classic definition of Theil’s statistic suffers from a number of limitations. In particular, a set of near zero preliminary estimates would always give a value of the U1 statistic close to 1 even though they are close to the latter estimates.

The second statistic, U2, is given by $$ U_2=\frac{\sqrt{\sum^n_{t=1}\left(\frac{P_{t + 1}-L_{t + 1}}{L_t}\right)^2}}{\sqrt{\sum^n_{t=1}\left(\frac{L_{t + 1}-L_t}{L_t}\right)^2}} $$ The interpretation of U2 differs from U1. The value of 1 is no longer the upper bound of the statistic but a threshold above (below) which the preliminary estimates is less (more) accurate than the naïve random walk forecast repeating the last observed value (considering P_t + 1 = L_t). Whenever it can be calculated (L_t ≠ 0 ∀t), the U2 statistic is the preferred option to evaluate the relevancy of the preliminary estimates.

T-test and Augmented T-test

To test whether the mean revision is statistically different from zero for a sample of n, we employ a conventional t-statistic $$ t=\frac{\overline{R}}{\sqrt{s^2/n}} $$ If the null hypothesis that the bias is equal to zero is rejected, it may give an insight of whether a bias exists in the earlier estimates.

The t-test is equivalent to fitting a linear regression of the revisions on only a constant (i.e. the mean). Assumptions such as the gaussianity of the revisions are implied. One can release the assumption of no autocorrelation by adding it into the model. Hence we have R_t = μ + ε_t,  t = 1, ..., n And the errors are thought to be serially correlated according to an AR(1) model, that is ε_t = γε_t + u_t,  with |γ| < 1 and u_t ∼ iid The auto-correlation in the error terms reduces the number of independent observations (and the degrees of freedom) by a factor of $n\frac{(1-\gamma)}{(1+\gamma)}$ and thus, the variance of the mean should be adjusted upward accordingly.

Hence, the Augmented t-test is calculated as $$ t_{adj}=\frac{\overline{R}}{\sqrt{\frac{s^2(1+\hat{\gamma})}{n(1-\hat{\gamma})}}} $$ where $$ \hat{\gamma}=\frac{\sum^{n-1}_{t=1}(R_t-\overline{R})(R_{t + 1}-\overline{R})}{\sum^n_{t=1}(R_t-\overline{R})^2} $$

Slope and drift

We assume a linear regression model of a latter vintage (L) on a preliminary vintage (P) and estimate the intercept (drift) β₀ and slope coefficient β₁ by OLS. The model is

L_t = β₀ + β₁P_t + ε_t While the (augmented) t-test on revisions only gives information about mean bias, this regression enables to assess both mean and regression bias. A regression bias would occur, for example, if preliminary estimates tend to be too low when latter estimates are relatively high and too high when latter estimates are relatively low. In this case, this may result in a positive value of the intercept and a value of β₁ < 1. To evaluate whether a mean or regression bias is present, we employ a conventional t-test on the parameters with the null hypothesis β₀ = 0 and β₁ = 1.

Recall that OLS regressions always come along with rather strict assumptions. Users should check the diagnostics and draw the necessary conclusions from them.

Regression of revisions on previous estimates

We assume a linear regression model of the revisions (R) on a preliminary vintage (P) and estimate the intercept β₀ and slope coefficient β₁ by OLS. The model is

R_t = β₀ + β₁P_t + ε_t,  t = 1, ..., n If revisions are affected by preliminary estimates, it means that those are not efficient and could be improved. We employ a conventional t-test on the parameters with the null hypothesis β₀ = 0 and β₁ = 0. Diagnostics on the residuals should be verified.

Regression of revisions from latter vintages on revisions from the previous vintages

We assume a linear regression model of the revisions between latter vintages (R_v) on the revisions between the previous vintages (R_v − 1) and estimate the intercept β₀ and slope coefficient β₁ by OLS. The model is

R_v, t = β₀ + β₁R_{v − 1, t} + ε_t,  t = 1, ..., n If latter revisions are predictable from previous revisions, it means that preliminary estimates are not efficient and could be improved. We employ a conventional t-test on the parameters with the null hypothesis β₀ = 0 and β₁ = 0. Diagnostics on the residuals should be verified.

Regression of latter revisions (Rv) on previous revisions (Rv_1, Rv_2,…Rv_p)

We assume a linear regression model of the revisions from latter vintages (R_v) on the revisions from p previous vintages (R_v − 1, ..., R_v − p) and estimate the intercept β₀ and slope coefficients of β₁, ..., β_p by OLS. The model is

$$ R_{v,t}=\beta_0+\sum^p_{i=1}\beta_{i}R_{v-i,t}+\varepsilon_t, ~~t=1,...,n $$ If latter revisions are predictable from previous revisions, it means that preliminary estimates are not efficient and could be improved. We employ a conventional t-test on the intercept parameter with the null hypothesis β₀ = 0 and a F-test to check the null hypothesis that β₁ = β₂ = ... = β_p = 0. Diagnostics on the residuals should be verified

Regression of latter revisions (Rv) on revisions from a specific vintage (Rv_k)

We assume a linear regression model of the revisions from latter vintages (R_v) on the revisions from a specific vintage (R_v − k) and estimate the intercept β₀ and slope coefficient β₁ by OLS. The model is

R_v, t = β₀ + β₁R_{v − k, t} + ε_t,  t = 1, ..., n If latter revisions are predictable from previous revisions, it means that preliminary estimates are not efficient and could be improved. We employ a conventional t-test on the parameters with the null hypothesis β₀ = 0 and β₁ = 0. Diagnostics on the residuals should be verified.

Test of autocorrelations in revisions

To test whether autocorrelation is present in revisions, we employ the Ljung-Box test. The Ljung-Box test considers together a group of autocorrelation coefficients up to a certain lag k and is therefore sometimes referred to as a portmanteau test. For the purpose of revision analysis, as we expect a relatively small number of observations, we decided to restrict the number of lags considered to k = 2. Hence, the users can also make the distinction with autocorrelation at seasonal lags (see seasonality tests below).

The null hypothesis is no autocorrelation. The test statistic is given by $$ Q=n(n+2)\sum^k_{i=1}\frac{\hat{\rho}_i^2}{n-i} $$ where n is the sample size, ρ̂_i² is the sample autocorrelation at lag in and k = 2 is the number of lags being tested. Under the null hypothesis, Q ∼ χ²(k).

If Q is statistically different from zero, the revision process may be locally biased in the sense that latter revisions are related to the previous ones.

Test of seasonality in revisions

To test whether seasonality is present in revisions, we employ two tests: the parametric QS test and the non-parametric Friedman test. Note that seasonality tests are always performed on first-differentiated series to avoid misleading results.

The QS test is a variant of the Ljung-Box test computed on seasonal lags, where we only consider positive auto-correlations $$ QS=n(n+2)\sum^k_{i=1}\frac{\left[max(0,\hat{\gamma}_{i.l})\right]^2}{n-i.l} $$ where k = 2, so only the first and second seasonal lags are considered. Thus, the test would checks the correlation between the actual observation and the observations lagged by one and two years. Note that l = 12 when dealing with monthly observations, so we consider the auto-covariances γ̂₁₂ and γ̂₂₄ alone. In turn k = 4 in the case of quarterly data.

Under the null hypothesis of no autocorrelation at seasonal lags, QS ∼ χ_modified²(k). The elimination of negative correlations calls indeed for a modified χ² distribution. This was done using simulation techniques.

The Friedman test requires no distributional assumptions. It uses the rankings of the observations. It is constructed as follows. Consider first the matrix of data {x_ij}_nxk with n rows (the blocks, i.e. the number of years in the sample), k columns (the treatments, i.e., either 12 months or 4 quarters, depending on the frequency of the data). The data matrix needs to be replaced by a new matrix {r_ij}_nxk, where the entry r_ij is the rank of x_ij within the block i. The test statistic is given by

$$ Q=\frac{n\sum^k_{j=1}(\tilde{r}_{.j}-\tilde{r})^2}{\frac{1}{n(k-1)}\sum^n_{i=1}\sum^k_{j=1}(r_{ij}-\tilde{r})^2} $$ The denominator represents the variance of the average ranking across treatments j relative to the total. Under the null hypothesis of no (stable) seasonality, Q ∼ χ²(k − 1).

As for non-seasonal autocorrelation tests at lower lags, if QS and Q are significantly different from zero, the revision process may be locally biased in the sense that latter revisions are related to the previous ones.