Title: | Interface Between the 'JDemetra+' Cruncher and R, and Quality Report Generator |
---|---|
Description: | Tool for generating quality reports from cruncher outputs (and calculating series scores). The latest version of the cruncher can be downloaded here: <https://github.com/jdemetra/jwsacruncher/releases>. |
Authors: | Tanguy Barthelemy [aut, cre, art], Alain Quartier-la-Tente [aut] , Institut national de la statistique et des études économiques [cph] (https://www.insee.fr/), Anna Smyk [aut] |
Maintainer: | Tanguy Barthelemy <[email protected]> |
License: | EUPL |
Version: | 0.2.4 |
Built: | 2024-09-06 06:09:14 UTC |
Source: | https://github.com/InseeFr/JDCruncheR |
Function to add indicators in QR_matrix
objects.
add_indicator(x, indicator, variable_name, ...)
add_indicator(x, indicator, variable_name, ...)
x |
a |
indicator |
a |
variable_name |
a string containing the name of the variables to add. |
... |
other parameters of the function |
The function add_indicator()
adds the chosen indicator to the values matrix of a quality report.
Therefore, because said indicator isn't added in the modalities matrix, it cannot be used to calculate a score (except for weighting).
Before using the added variable for score calculation, it will have to be coded with the function recode_indicator_num
.
The new indicator can be a vector
or a data.frame
. In both cases, its format must allow for pairing:
a vector
's elements must be named and these names must match those of the quality report (variable "series");
a data.frame
must contain a "series" column that matches with the quality report's series.
This function returns the same object, enhanced with the chosen indicator.
So if the input x
is a QR_matrix, an object of class QR_matrix is returned.
If the input x
is a mQR_matrix, an object of class mQR_matrix is returned.
Other var QR_matrix manipulation:
QR_var_manipulation
,
recode_indicator_num()
To calculate a score for each series from a quality report
## S3 method for class 'QR_matrix' compute_score( x, score_pond = c(qs_residual_sa_on_sa = 30, f_residual_sa_on_sa = 30, qs_residual_sa_on_i = 20, f_residual_sa_on_i = 20, f_residual_td_on_sa = 30, f_residual_td_on_i = 20, oos_mean = 15, oos_mse = 10, residuals_independency = 15, residuals_homoskedasticity = 5, residuals_skewness = 5, m7 = 5, q_m2 = 5), modalities = c("Good", "Uncertain", "", "Bad", "Severe"), normalize_score_value, na.rm = FALSE, n_contrib_score, conditional_indicator, ... )
## S3 method for class 'QR_matrix' compute_score( x, score_pond = c(qs_residual_sa_on_sa = 30, f_residual_sa_on_sa = 30, qs_residual_sa_on_i = 20, f_residual_sa_on_i = 20, f_residual_td_on_sa = 30, f_residual_td_on_i = 20, oos_mean = 15, oos_mse = 10, residuals_independency = 15, residuals_homoskedasticity = 5, residuals_skewness = 5, m7 = 5, q_m2 = 5), modalities = c("Good", "Uncertain", "", "Bad", "Severe"), normalize_score_value, na.rm = FALSE, n_contrib_score, conditional_indicator, ... )
x |
a |
score_pond |
the formula used to calculate the series score. |
modalities |
modalities ordered by importance in the score calculation (cf. details). |
normalize_score_value |
integer indicating the reference value for weights normalisation. If missing, weights will not be normalised. |
na.rm |
logical indicating whether missing values must be ignored when calculating the score. |
n_contrib_score |
integer indicating the number of variables to create in the quality report's values matrix
to store the |
conditional_indicator |
a |
... |
other unused parameters. |
The function compute_score
calculates a score from the modalities of a quality report: to each modality corresponds
a weight that depends on the parameter modalities
. The default parameter is c("Good", "Uncertain", "Bad","Severe")
,
and the associated weights are respectively 0, 1, 2 and 3.
The score calculation is based on the score_pond
parameter, which is a named integer vector containing the weights
to apply to the (modalities matrix) variables. For example, with score_pond = c(qs_residual_sa_on_sa = 10, f_residual_td_on_sa = 5)
,
the score will be based on the variables qs_residual_sa_on_sa and f_residual_td_on_sa.
The qs_residual_sa_on_sa grades will be multiplied by 10 and the f_residual_td_on_sa grades, by 5.
To ignore the missing values when calculating a score, use the parameter na.rm = TRUE
.
The parameter normalize_score_value
can be used to normalise the scores. For example, to have all scores between 0 and 20,
specify normalize_score_value = 20
.
When using parameter n_contrib_score
, n_contrib_score
new variables are added to the quality report's values matrix.
These new variables store the names of the variables that contribute the most to the series score.
For example, n_contrib_score = 3
will add to the values matrix the three variables that contribute the most to the score.
The new variables' names are i_highest_score, with i being the rank in terms of contribution to the score (1_highest_score
contains the name of the greatest contributor, 2_highest_score the second greatest, etc).
Only the variables that have a non-zero contribution to the score are taken into account: if a series score is 0,
all i_highest_score variables will be empty. And if a series score is positive only because of the m7 statistic,
1_highest_score will have a value of "m7" for this series and the other i_highest_score will be empty.
Some indicators are only relevant under certain conditions. For example, the homoscedasticity test is only valid when the residuals are independant, and the normality tests,
only when the residuals are both independant and homoscedastic. In these cases, the parameter conditional_indicator
can be of use
since it reduces the weight of some variables down to 1 when some conditions are met.
conditional_indicator
is a list
of 3-elements sub-lists:
"indicator": the variable whose weight will be conditionally changed
"conditions": the variables used to define the conditions
"conditions_modalities": modalities that must be verified to induce the weight change
For example, conditional_indicator = list(list(indicator = "residuals_skewness", conditions = c("residuals_independency", "residuals_homoskedasticity"), conditions_modalities = c("Bad","Severe")))
,
reduces down to 1 the weight of the variable "residuals_skewness" when the modalities of the independancy test ("residuals_independency")
or the homoscedasticity test ("residuals_homoskedasticity") are "Bad" or "Severe".
a QR_matrix
or mQR_matrix
object.
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Calculer le score QR <- compute_score(QR, n_contrib_score = 2) print(QR) # Extract the modalities matrix: QR$modalities$score
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Calculer le score QR <- compute_score(QR, n_contrib_score = 2) print(QR) # Extract the modalities matrix: QR$modalities$score
Exporting QR_matrix or mQR_matrix objects in an Excel file
export_xlsx(x, ...)
export_xlsx(x, ...)
x |
a |
... |
other parameters of the function |
If x
is a mQR_matrix
, the function returns invisibly (via invisible(x)
) the same mQR_matrix
object as x
.
Else if x
is a QR_matrix
, the function returns invisibly (via invisible(x)
) a workbook object created by XLConnect::loadWorkbook()
for further manipulation.
Other QR_matrix functions:
export_xlsx.QR_matrix()
,
export_xlsx.mQR_matrix()
,
extract_QR()
,
rbind.QR_matrix()
,
sort()
,
weighted_score()
To export several quality reports in Excel files
## S3 method for class 'mQR_matrix' export_xlsx( x, export_dir, layout_file = c("ByComponent", "ByQRMatrix", "AllTogether"), auto_format = TRUE, overwrite = TRUE, ... )
## S3 method for class 'mQR_matrix' export_xlsx( x, export_dir, layout_file = c("ByComponent", "ByQRMatrix", "AllTogether"), auto_format = TRUE, overwrite = TRUE, ... )
x |
a |
export_dir |
export directory. |
layout_file |
export parameter. By default,
( |
auto_format |
logical indicating whether to format the output
( |
overwrite |
logical indicating whether to create an Excel file if it
doesn't exist yet ( |
... |
other unused arguments |
Returns invisibly (via invisible(x)
) the same mQR_matrix
object as x
.
Other QR_matrix functions:
export_xlsx()
,
export_xlsx.QR_matrix()
,
extract_QR()
,
rbind.QR_matrix()
,
sort()
,
weighted_score()
To export a quality report in an Excel file.
## S3 method for class 'QR_matrix' export_xlsx(x, file, auto_format = TRUE, overwrite = TRUE, ...)
## S3 method for class 'QR_matrix' export_xlsx(x, file, auto_format = TRUE, overwrite = TRUE, ...)
x |
a |
file |
a |
auto_format |
logical indicating whether to format the output
( |
overwrite |
logical indicating whether to create an Excel file if it
doesn't exist yet ( |
... |
other unused arguments |
Returns invisibly (via invisible(x)
) a workbook object created by XLConnect::loadWorkbook()
for further manipulation.
Other QR_matrix functions:
export_xlsx()
,
export_xlsx.mQR_matrix()
,
extract_QR()
,
rbind.QR_matrix()
,
sort()
,
weighted_score()
To extract a quality report from the csv file containing the diagnostics matrix.
extract_QR(matrix_output_file, sep = ";", dec = ",")
extract_QR(matrix_output_file, sep = ";", dec = ",")
matrix_output_file |
the csv file containing the diagnostics matrix. |
sep |
the separator used in the csv file (by default, |
dec |
the decimal separator used in the csv file (by default, |
This function generates a quality report from a csv file containing diagnostics (usually from the file demetra_m.csv).
The demetra_m.csv file can be generated by launching the cruncher (functions cruncher
or cruncher_and_param
) with the default
export parameters, having used the default option csv_layout = "vtable"
to format the output tables of the functions
cruncher_and_param
and create_param_file
when creating the parameters file.
This function returns a QR_matrix
object, which is a list of 3 objects:
modalities
, a data.frame
containing several indicators and their categorical quality (Good, Uncertain, Bad, Severe).
values
, a data.frame
containing the same indicators and the values that lead to their quality category (i.e.: p-values, statistics, etc.)
as well as additional variables that don't have a modality/quality (series frequency and arima model).
score_formula
that will store the formula used to calculate the score (when relevant). Its initial value is NULL
.
a QR_matrix
object.
Other QR_matrix functions:
export_xlsx()
,
export_xlsx.QR_matrix()
,
export_xlsx.mQR_matrix()
,
rbind.QR_matrix()
,
sort()
,
weighted_score()
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) print(QR) # Extract the modalities matrix: QR$modalities # Or: QR[["modalities"]]
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) print(QR) # Extract the modalities matrix: QR$modalities # Or: QR[["modalities"]]
To extract score variables from QR_matrix
or mQR_matrix
objects.
extract_score( x, format_output = c("data.frame", "vector"), weighted_score = FALSE )
extract_score( x, format_output = c("data.frame", "vector"), weighted_score = FALSE )
x |
a |
format_output |
string of characters indicating the output format: either a |
weighted_score |
logical indicating whether to extract the weighted score (if previously calculated) or the unweighted one. By default, the unweighted score is extracted. |
For QR_matrix
objects, the output is a vector or the object NULL
if no score was previously calculated.
For mQR_matrix
objects, it is a list of scores (NULL
elements or vectors).
extract_score()
returns a data.frame with two column: the series name and their score.
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR1 <- compute_score(x = QR, n_contrib_score = 5) QR2 <- compute_score( x = QR, score_pond = c(qs_residual_sa_on_sa = 5, qs_residual_sa_on_i = 30, f_residual_td_on_sa = 10, f_residual_td_on_i = 40, oos_mean = 30, residuals_skewness = 15, m7 = 25) ) mQR <- mQR_matrix(list(a = QR1, b = QR2)) # Extract score extract_score(QR1) extract_score(mQR)
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR1 <- compute_score(x = QR, n_contrib_score = 5) QR2 <- compute_score( x = QR, score_pond = c(qs_residual_sa_on_sa = 5, qs_residual_sa_on_i = 30, f_residual_td_on_sa = 10, f_residual_td_on_i = 40, oos_mean = 30, residuals_skewness = 15, m7 = 25) ) mQR <- mQR_matrix(list(a = QR1, b = QR2)) # Extract score extract_score(QR1) extract_score(mQR)
To print information on a QR_matrix or mQR_matrix object.
## S3 method for class 'QR_matrix' print(x, print_variables = TRUE, print_score_formula = TRUE, ...) ## S3 method for class 'mQR_matrix' print(x, score_statistics = TRUE, ...)
## S3 method for class 'QR_matrix' print(x, print_variables = TRUE, print_score_formula = TRUE, ...) ## S3 method for class 'mQR_matrix' print(x, score_statistics = TRUE, ...)
x |
a |
print_variables |
logical indicating whether to print the indicators' name (including additionnal variables). |
print_score_formula |
logical indicating whether to print the formula with which the score was calculated (when calculated). |
... |
other unused arguments. |
score_statistics |
logical indicating whether to print the statistics in the |
the print
method prints a mQR_matrix
or mQR_matrix
object and returns it invisibly (via invisible(x)
).
mQR_matrix()
and QR_matrix()
are creating one (or several) quality report. The function
is.QR_matrix()
and is.mQR_matrix()
are functions to test whether an object is a quality report or a list of quality reports.
QR_matrix(modalities = NULL, values = NULL, score_formula = NULL) mQR_matrix(x = list(), ...) is.QR_matrix(x) is.mQR_matrix(x)
QR_matrix(modalities = NULL, values = NULL, score_formula = NULL) mQR_matrix(x = list(), ...) is.QR_matrix(x) is.mQR_matrix(x)
modalities |
a |
values |
a |
score_formula |
the formula used to calculate the series score (if defined). |
x |
a |
... |
objects of the same type as |
AQR_matrix
object is a list of three items:
modalities
, a data.frame
containing a set of categorical variables (by default: Good, Uncertain, Bad, Severe).
values
, a data.frame
containing the values corresponding to the modalities
indicators (i.e. p-values, statistics, etc.),
as well as variables for which a modality cannot be defined (e.g. the series frequency, the ARIMA model, etc).
score_formula
contains the formula used to calculate the series score (once the calculus is done).
QR_matrix()
creates and returns a QR_matrix
object.
mQR_matrix()
creates and returns a mQR_matrix
object (ie. a list of QR_matrix
objects).
is.QR_matrix()
and is.mQR_matrix()
return Boolean values (TRUE
or FALSE
).
Functions to remove indicators (remove_indicators()
) or retrain some indicators only (retain_indicators()
)
from QR_matrix
or mQR_matrix
objects. The series names (column "series") cannot be removed.
remove_indicators(x, ...) retain_indicators(x, ...)
remove_indicators(x, ...) retain_indicators(x, ...)
x |
a |
... |
names of the variable to remove (or keep) |
remove_indicators()
returns the same object x
reduced by the flags and variables used as arguments ...
So if the input x
is a QR_matrix, an object of class QR_matrix is returned.
If the input x
is a mQR_matrix, an object of class mQR_matrix is returned.
Other var QR_matrix manipulation:
add_indicator()
,
recode_indicator_num()
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR <- compute_score(QR, n_contrib_score = 2) # Retain indicators retain_indicators(QR, "score", "m7") # retaining "score" and "m7" retain_indicators(QR, c("score", "m7")) # Same # Remove indicators QR <- remove_indicators(QR, "score") # removing "score" extract_score(QR) # is NULL because we removed the score indicator
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR <- compute_score(QR, n_contrib_score = 2) # Retain indicators retain_indicators(QR, "score", "m7") # retaining "score" and "m7" retain_indicators(QR, c("score", "m7")) # Same # Remove indicators QR <- remove_indicators(QR, "score") # removing "score" extract_score(QR) # is NULL because we removed the score indicator
Function to combine multiple QR_matrix
objects: line by line, both for the modalities
and the values
table.
## S3 method for class 'QR_matrix' rbind(..., check_formula = TRUE)
## S3 method for class 'QR_matrix' rbind(..., check_formula = TRUE)
... |
|
check_formula |
logical indicating whether to check the score formulas' coherency.
By default, |
rbind.QR_matrix()
returns a QR_matrix
object.
Other QR_matrix functions:
export_xlsx()
,
export_xlsx.QR_matrix()
,
export_xlsx.mQR_matrix()
,
extract_QR()
,
sort()
,
weighted_score()
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute differents scores QR1 <- compute_score(QR, score_pond = c(m7 = 2, q = 3, qs_residual_sa_on_sa = 5)) QR2 <- compute_score(QR, score_pond = c(m7 = 2, qs_residual_sa_on_sa = 5)) # Merge two quality report try(rbind(QR1, QR2)) # Une erreur est renvoyée rbind(QR1, QR2, check_formula = FALSE)
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute differents scores QR1 <- compute_score(QR, score_pond = c(m7 = 2, q = 3, qs_residual_sa_on_sa = 5)) QR2 <- compute_score(QR, score_pond = c(m7 = 2, qs_residual_sa_on_sa = 5)) # Merge two quality report try(rbind(QR1, QR2)) # Une erreur est renvoyée rbind(QR1, QR2, check_formula = FALSE)
To transform variables from the values matrix into categorical variables that can be added into the modalities matrix.
recode_indicator_num( x, variable_name, breaks = c(0, 0.01, 0.05, 0.1, 1), labels = c("Good", "Uncertain", "Bad", "Severe"), ... )
recode_indicator_num( x, variable_name, breaks = c(0, 0.01, 0.05, 0.1, 1), labels = c("Good", "Uncertain", "Bad", "Severe"), ... )
x |
a |
variable_name |
a vector of strings containing the names of the variables to convert. |
breaks |
see function |
labels |
see function |
... |
other parameters of the |
The function recode_indicator_num()
returns the same object, enhanced with the chosen indicator.
So if the input x
is a QR_matrix, an object of class QR_matrix is returned.
If the input x
is a mQR_matrix, an object of class mQR_matrix is returned.
Other var QR_matrix manipulation:
QR_var_manipulation
,
add_indicator()
To sort the quality reports on one or several variables
## S3 method for class 'QR_matrix' sort(x, decreasing = FALSE, sort_variables = "score", ...) ## S3 method for class 'mQR_matrix' sort(x, decreasing = FALSE, sort_variables = "score", ...)
## S3 method for class 'QR_matrix' sort(x, decreasing = FALSE, sort_variables = "score", ...) ## S3 method for class 'mQR_matrix' sort(x, decreasing = FALSE, sort_variables = "score", ...)
x |
a |
decreasing |
logical indicating whether the quality reports must be sorted in ascending or decreasing order. By default, the sorting is done in ascending order. |
sort_variables |
They must be present in the modalities table. |
... |
other parameters of the function |
the input with sorted quality reports
Other QR_matrix functions:
export_xlsx()
,
export_xlsx.QR_matrix()
,
export_xlsx.mQR_matrix()
,
extract_QR()
,
rbind.QR_matrix()
,
weighted_score()
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR <- compute_score(QR, n_contrib_score = 2) print(QR$modalities$score) # Sort the scores QR <- sort(QR, sort_variables = "score") # Pour trier par ordre croissant sur le score print(QR$modalities$score)
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR <- compute_score(QR, n_contrib_score = 2) print(QR$modalities$score) # Sort the scores QR <- sort(QR, sort_variables = "score") # Pour trier par ordre croissant sur le score print(QR$modalities$score)
Function to weight a pre-calculated score
weighted_score(x, pond = 1)
weighted_score(x, pond = 1)
x |
a |
pond |
the weights to use. Can be an integer, a vector of integers, the name of one of the quality report variables
or a list of weights for the |
the input with an additionnal weighted score
Other QR_matrix functions:
export_xlsx()
,
export_xlsx.QR_matrix()
,
export_xlsx.mQR_matrix()
,
extract_QR()
,
rbind.QR_matrix()
,
sort()
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR <- compute_score(QR, n_contrib_score = 2) # Weighted score QR <- weighted_score(QR, 2) print(QR) # Extract the weighted score QR$modalities$score_pond
# Path of matrix demetra_m demetra_path <- file.path( system.file("extdata", package = "JDCruncheR"), "WS/ws_ipi/Output/SAProcessing-1", "demetra_m.csv" ) # Extract the quality report from the demetra_m file QR <- extract_QR(demetra_path) # Compute the score QR <- compute_score(QR, n_contrib_score = 2) # Weighted score QR <- weighted_score(QR, 2) print(QR) # Extract the weighted score QR$modalities$score_pond