A4: How to assess current organic market data production and dissemination systems

Quality dimensions were defined by the Statistical office of the European Union (Eurostat) to establish a framework for the analysis and evaluation of the quality of statistical data and its sources: relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability, and coherence (Eurostat, 2009).

These quality dimensions are explained in more detail in the European Statistics Code of Practice (CoP), which presents the desired structure and content of a quality report to harmonise quality reporting across member states and to facilitate comparisons (Eurostat, 2009). In addition, the European Statistical System Committee prepared a Quality Assurance Framework to explain activities, methods and tools that help to implement the CoP (Eurostat, 2012). In the following paragraphs, each data quality dimension is explained by a number of aspects and questions which should be taken into account when assessing data quality. The introduction of a quality report has to include: a brief history of the statistical process and outputs in question; a main text body on statistics to which the outputs belong; and limitations of the quality report with references to related reports (Hahn & Linden, 2007).

The first quality dimension is relevance. It is defined as the degree to which statistical outputs meet current and potential user needs. To further describe the relevance of the statistical output it is necessary to refer to its contents and to provide the key outputs/estimates desired by different users.

The second quality dimension, accuracy, implies the closeness of data to the true values and covers sampling as well as non-sampling errors which have been explained above. The method of data collection (e.g. online survey) that is used needs to be presented in order to understand and assess specific errors. Furthermore, a section on the main sources of random and systematic errors needs to be provided. Depending on the type of study, particular errors have to be defined in more detail and have to be handled individually. In addition to sampling errors, one can find coverage, measurement, nonresponse, and data handling errors as described earlier in this chapter.

The third dimension consists of two quality indicators: namely timeliness and punctuality. Timeliness is defined as the length of time between the date to which the data refer and their availability for the public. Punctuality means the time lag between release date and target release date of data. The reasons for non-punctual releases need to be explained.

Accessibility and clarity comprise simplicity and ease with which users can access statistics. The conditions of data access depend on the following factors: media, support, pricing policies, and possible restrictions. The understanding of statistical outputs can be enhanced by the description of accompanying information. The best way to evaluate this quality dimension is a reflection on the feedback of users, which is an essential part of the quality report.

The two remaining dimensions refer to coherence and comparability of statistical data. The quality of statistical outputs depends on the use of the same concepts and harmonised methods. In this context comparability is defined as a special case of coherence. A lack of coherence is explained by differences in concepts and methods. Hence one part of the quality section needs to deal with the assessment of possible effects of each reported difference on the output values. To further explain this quality dimension it can be related to a variety of attributes. First of all, comparability can be regarded over time and across regions; secondly coherence can be evaluated internally, but also in comparison with national accounts or with other statistics; and finally the quality can be checked with the help of so-called mirror statistics, which usually tackle the same topic but use a different sample or a different method (Eurostat, 2009).

Eurostat (2009) has also published a list of Standard Quality Indicators for each dimension of data quality.

To assess (or self-assess) organic market data production and dissemination systems, we have slightly adapted the Eurostat methodology and indicators (Table A.4-1).

Table A.4-1: Quality and performance indicators to assess current organic market data