Overview
This project stems from the dialogue between the EOSC Landscape WG and the INFRAEOSC5 project's landscaping task force about the necessity to move from static landscapes to dynamic monitoring of EOSC readiness indicators for MS. After having consulted with diverse EOSC stakeholders and having proposed an initial set of indicators, the task force decided to obtain specialised consultancy to develop a Proof of Concept (PoC) for a platform monitoring such live indicators.
The main objective was to validate the possibility of developing a dashboard solution, that would minimise the effort needed to collect the information. The dashboard would integrate data harvested from trusted open data sources automatically with the information provided manually by delegates from the MS, and facilitate the semi-automated evaluation of EOSC readiness against indicators.
The project ran from February to September 2021. It was carried out by using an iterative co-creation approach, that covered the different phases from the collection of requirements to the study of national use cases to the validation of the results and final recommendations. The experts who carried out the PoC liaised closely with the INFRAEOSC5 landscaping task force and were involved in the co-creation of several groups of stakeholders: National OS initiatives, Data/service providers, relevant projects and initiatives, RI and e-Infrastructure representatives, cadres of the EOSC Association, delegates in the EOSC Steering Board, and EC officers.
The project included three different aspects:
• the analysis of the 18 EOSC readiness indicators proposed by the INFRAEOSC Landscaping TF identifying possible data sources and assessing data availability, completeness and updates;
• the co-design and testing of a possible dashboard model, to define key functionalities and use cases, design the process to practically implement the monitoring process and the actors involved;
• the PoC technical development and testing of technical features and usability aspects.
The findings were presented and validated at a final validation workshop, together with a list of recommendations to further progress the monitoring work.
Objectives & Challenges
The objectives of the project were:
- Use the initial list of indicators and related metrics defined by the Landscape Task Force, and reuse relevant outcomes produced by INFRAEOSC-5b and predecessor projects.
- Design the dashboard and select an appropriate tool to implement it. Define the workflows for automatic and non-automatic data collection.
- Define processes for practically implementing the active monitoring, reporting and analysis of progress against the indicators, including user guidelines.
- Define mechanisms to ensure the transparency, accountability and actionability of the collected information.
- Ensure flexible solutions to update and develop new indicators.
- Involve beta testers among the NOSCIs/Mandated Organizations to test and tune the proposed solutions.
- Arrange a virtual Validation Workshop to present the work done and collect additional feedback to consolidate the PoC results.
- Provide an estimate of the effort and any other requirements (e.g. software compliance with standards) needed to implement the monitoring on an annual basis at the country level.
Main Findings
The project assessed the availability and robustness of open data sources for the proposed indicators, highlighting a general gap in the availability of trusted, up-to-date, complete data that are comparable across countries that limits, in the short term, the possibility to implement automated or semi-automated monitoring of the indicators at the European level. On the other hand, the PoC also showed a very high level of interest among the stakeholders in the development of such a dashboard, and positive feedback on the specific design proposed that suggest to re-use the collection of requirements and list of features, if not the specific technical solution, in further developments. Some key concepts were also identified that should be taken into account for further development: in particular, the need for data validation by MS, and the need to monitor trends and signs of progress.
Data availability and data trustworthiness are key elements in further developing an EOSC readiness dashboard, and at present, they are two major challenges for the full implementation of a dashboard. Nevertheless, the approach followed in the PoC is considered viable and interesting by most of the stakeholders meet during the project. Moreover, several national and international organisations are working on indicators definition and data repository development, first of all, national registries for EOSC services and training.
Much remains to do in this respect and as a part of the work, we provided recommendations on how to progress in the short, medium and long term.
Main Recommendations
The project drafted short and medium-to-long-term recommendations on how to further the idea of an EOSC readiness dashboard, that will feed in relevant activities, first of all, the EOSC Observatory, being now developed in the framework of the EOSC-future project.
A key recommendation that emerges from our findings is that PoC clearly showed that the development of an EOSC readiness platform is a socio-technical task and, if from a technological point of view, there are various efficient Open Source solutions (such as the ones used in the PoC), the “social” aspects of the work are more complex and demanding. More specifically, the work on indicator selection, data availability analysis, data quality assurance, data elaboration, data entry governance design, data visualisation and user interphase design ask for the active engagement of social scientists in the development team. Indeed, experts of sociology of science and innovation, co-design and participatory processes and statisticians should be part of such a team.
