Explanatory Memorandum to COM(2013)5 - Interim evaluation of the ISA programme

Please note

This page contains a limited version of this dossier in the EU Monitor.

dossier COM(2013)5 - Interim evaluation of the ISA programme.
source COM(2013)5 EN
date 18-01-2013
REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Interim evaluation of the ISA programme

1. Introduction

By decision of the European Parliament and the Council (the ISA Decision)1 the six-year programme on interoperability solutions for European public administrations (the ISA programme) was launched on 1 January 2010 as a follow-up programme to the IDABC programme2.

The purpose of this report is to present the findings and recommendations of the interim evaluation of the ISA programme. The need to carry out the evaluation stems from Article 13(3) of the ISA Decision, which also requires the European Commission to communicate the results of the evaluation to the European Parliament and the Council by 31 December 2012.

The evaluation3 was performed by the Commission using a team of independent experts from a consultancy company (the evaluation team). Representatives of Commission services have overseen the evaluation through a Commission steering group4.

2. Background

The objective of the ISA programme is to support cooperation between European public administrations by facilitating the efficient and effective electronic cross-border and cross-sectoral interaction between such administrations, including bodies performing public functions on their behalf, with a view to enabling the delivery of electronic public services supporting the implementation of EU policies and activities5.

As defined, the objective covers both the ultimate goal of supporting cooperation between European public administrations with a view to enabling the delivery of electronic public services supporting the implementation of EU policies and activities and the intermediate objective of facilitating efficient and effective electronic cross-border and cross-sectoral interaction between European public administrations to this end.

To meet both the ultimate and intermediate objectives and to provide ‘common and shared solutions facilitating interoperability’6, the ISA programme launches studies, projects and accompanying measures, in specific actions, to support the

- Establishment and improvement of common frameworks in support of cross-border and cross-sectoral interoperability,

- Operation and improvement of existing common services as well as the establishment of new ones,

- Improvement of existing reusable generic tools as well as the establishment of new ones,

- Assessment of the ICT implications of EU legislation.

Common frameworks are strategies, specifications, methodologies, guidelines and similar approaches and documents.

Common services are operational applications and infrastructures of a generic nature which meet common user requirements across policy areas.

Generic tools are reference platforms, shared and collaborative platforms, common components and similar building blocks that meet common user requirements across policy areas.

Specific actions funded by the programme are listed in the rolling ISA Work Programme, which the Commission amends at least once a year after consulting the ISA Committee, established by the ISA Decision.

3. Methodology

The interim evaluation of the ISA programme relied on multiple methods of investigation for collecting quantitative and qualitative data, namely desk research, online surveys, interviews, written requests for additional information and case studies, involving a broad range of representatives from Member States and EU services as well as a limited number of other stakeholders.

4. Evaluation issues and questions

Contents

1.

The evaluation focused on the following six main criteria:


- Relevance — to what extent do the actions funded by the ISA programme contributes to the achievement of the programme objective(s)? To what extent are the ISA programme’s objective(s) still pertinent in relation to evolving needs and priorities at both national and EU level?

- Efficiency — how efficiently are the various inputs and actions being converted into outputs and results? What aspects of the programme are the most efficient or inefficient, especially in terms of resources mobilised?

- Effectiveness — how effective have the ISA programme’s results and impacts been in achieving its specific and general objectives? Are there aspects that are more or less effective than others, and — if so — what lessons can be drawn from this?

- Utility — how do the ISA programme’s actions, results and impacts, achieved and anticipated, compare with the needs they are supposed to address? To what extent could measures be taken to improve the utility of the ISA programme’s actions, and what measures would these be?

- Sustainability — to what extent is the financial, technical and operational sustainability of solutions ensured?

- Coherence — to what extent do the actions form part of a ‘holistic’ approach within the programme framework? How well are synergies achieved between programme actions and with other EU activities?

Apart from the specific questions linked to the evaluation criteria, a number of related questions were raised. Furthermore, during the evaluation the Commission steering group added an additional criterion:

— Coordination — to what extent are activities coordinated or aligned with the needs of other stakeholders and Member States? To what extent are the activities under other EU initiatives coordinated with ISA actions?

5. Findings

2.

Based on the data collected, the evaluation team has presented its findings in relation to the evaluation issues and questions defined. They may be summarised as follows:


5.1. Relevance

With a view to the ultimate objective of the ISA programme, the Commission has demonstrated that the programme is fully aligned with the intermediate objective, as 88 % of the ISA actions facilitate efficient and effective cross-sectoral interaction between European public administrations and 94 % facilitate electronic cross-border interaction between public administrations across Europe. However, the cross-sectoral aspect of the intermediate ISA objective was not mentioned in the majority of the interviews and survey responses.

The ISA programme is closely aligned with both the Digital Agenda for Europe and the eGovernment Action Plan 2011-2015 and is perceived as contributing to the implementation of the Single Market Act, the Service Directive, the environmental protection directives and the Reuse of Public Sector Information Directive.

In addition, the programme is considered to be relevant in responding to Member States’ needs, as the needs addressed are still recognised as current needs of European public administrations.

The improvement of coordination between the ISA programme and both the Member States and Commission services and the increased reusability of ISA solutions were additional needs identified by Member States.

5.2. Efficiency

Overall, the ISA programme is considered to be efficient. First of all, the smooth process leading to the ISA Committee’s favourable opinions on the ISA rolling work programme and its first and second revisions, as well as the swift release of funds on average six working days after adoption of the work programme and its revisions, have ensured efficient implementation. Secondly, as the budget allocated is almost equal to the budget committed, and the total budget for the period 2010-12 is very close to the estimated financial cost for that period, the budgetary resources are considered to be allocated and consumed efficiently. As to programme performance, it is generally on track, despite some delays in meeting some milestones

Some aspects of the ISA programme are not perceived as efficient, two in particular being singled out. Firstly, the engagement of stakeholders is not considered efficient, in particular the Member States’ involvement in the programme, more specifically that of national interoperability experts, which can partly be explained by a lack of resources at national level. Second, the allocation of human resources is considered by the evaluators as only partially adequate, given the high turnover and the distribution of resources compared with the figures specified in the financial statement accompanying the proposal for the ISA Decision when it was adopted by the Commission and transmitted to the European Parliament and the Council. Some interviews pointed to the lack of expertise in specific domains, a lack of budgetary resources and inefficiency in implementing the programme, but the evaluators found no evidence confirming these perceptions.

5.3. Effectiveness

A specific assessment of the results achieved by the programme is not yet feasible as the ISA programme is still in its early stages and very few new actions have so far delivered results. Common frameworks were perceived as the most effective solutions provided by the programme as compared to common services or reusable generic tools. The action ‘Assessment of ICT implications’ was highlighted in particular as not yet having delivered the expected benefits.

5.4. Utility

Assessing the utility of a programme at the time of an interim evaluation is questionable and, in fact, this is normally only done in final evaluations. The evaluators have, nevertheless, addressed the reusability aspect of ISA solutions.

The results of the ISA programme are partially reused at both Member State and Commission level. The majority of Member State stakeholders stated that ISA solutions were being reused at Member State level. Although concrete solutions could not always be named, some 24 % of the ISA solutions are said to have been reused, representing roughly 65 % of the total budget of the programme, with sTesta accounting for the major share. A representative number of interviewees considered that ISA solutions are being reused in the Commission, although this perception is not shared by a small number of respondents to the online surveys, possibly indicating a lack of awareness of the ISA solutions.

5.5. Sustainability

As with utility, sustainability is typically not assessed during an interim evaluation. Even though the evaluators addressed this aspect, no definitive conclusion is drawn regarding the sustainability of the ISA solutions for two main reasons. First, the ISA programme is still in its early stages and has not yet yielded enough results and impacts that might be sustained. Second, as very little evidence could be gathered, the validity of any conclusion would be highly questionable.

The evaluation report noted the need to identify possible means to ensure the long term sustainability of an increasing number of solutions developed by the ISA programme (and its predecessors).

Regarding financial sustainability the proposed Connecting Europe Facility (CEF) was identified by the evaluators as one of the possible means of sustainability of some of the operational activities of ISA.

5.6. Coherence

Synergies are fairly well established within the ISA programme, as 78 % of the ISA actions have yielded internal synergies. More generally, the majority of stakeholders perceive synergies as not well or not well enough established or leading to some overlaps, which some stakeholders attribute to the lack of information communicated to stakeholders, the lack of synergies identified in the programme and the lack of a control mechanism to ensure the reuse of ISA solutions.

On the other hand, the ISA programme is achieving synergies with other EU initiatives. In fact, some ISA actions have synergies with the ICT Policy Support Programme under the Competitiveness and Innovation Framework Programme, the Open Data Portal and eParticipation, and to a lesser extent, with the ICT programme under the seventh Framework Programme (FP7-ICT), ICT standardisation and the European eJustice action plan.

5.7. Coordination

Member States are not sufficiently engaged in the ISA programme. Despite their involvement in defining the priorities of the programme, a work programme revision process that includes Member States, and the high participation of Member States in the ISA Committee and Coordination Group7 meetings, the interaction between the ISA programme and Member States to ensure that the ISA programme is implemented in line with the needs and priorities of the Member States is not perceived to be sufficiently effective. In the opinion of the evaluation team, the lack of involvement at national level can explain this gap and also explain the low level of awareness and reuse of the ISA solutions across Member States.

Coordination between the ISA programme and other EU initiatives is ensured through various means: formal and informal meetings, the Commission’s IT governance and its inter-service consultations, presentations to the CTI (‘Comité Technique Informatique’), communication activities and external events. As regards the perception among Commission services of the effectiveness of this coordination, a small majority of interviewees considered it not sufficiently effective to prevent duplication of work. They perceive this ineffective coordination as a barrier to raising awareness of the ISA solutions across Member States and Commission services, and, consequently, to the reuse of ISA solutions by these stakeholders.


6. Recommendations

The evaluation was largely positive, describing the ISA programme as aligned with the policy priorities of the European Commission and the needs of Member States and implemented efficiently and coherently, delivering results that are reused by both Commission services and Member States. Nevertheless the evaluation report also highlights some shortcomings and makes recommendations with regard to:

3.

- Communication and raising awareness


4.

- Engagement of stakeholders and project management continuity


5.

- Avoiding overlaps and duplications, of work, increasing reusability and ensuring sustainability


Furthermore, the evaluators consider that four of the eleven recommendations from the final evaluation of the predecessor programme IDABC are still applicable to the ISA programme.

6.1. Communication and raising awareness

The ISA programme must ensure that all stakeholders involved in the ISA programme are well aware of the objectives of each action, the contribution of these actions to the programme’s objectives, and the intended and actual results. Even though these aspects are covered in the work programme documentation and on the ISA portal, they should be made available in a simplified form to communicate more effectively to a wider audience. In addition, the cross-sector aspects of the ISA actions should be described more precisely8.

The ISA programme should conduct regular meetings with all action owners to exchange information on the current progress of actions and explore potential synergies9.

The ISA programme should reinforce promotion and communication activities regarding the ISA solutions that have produced concrete results by continuously participating in events at national level, organising ISA events, and issuing publications on these solutions, as well as by playing an active role in other EU activities or programmes supporting opportunities for the reuse of ISA solutions.

In line with the evaluators' recommendations, the ISA programme will increase collaboration with other stakeholders, i.e. other European institutions and the ICT industry. Furthermore, taking into consideration the evaluators recommendation it will identify relevant contacts within academia and private organisations that could bring added value through their involvement in specific ISA actions 10.

The Commission will analyse the cost-effectiveness of these recommendations.

The ISA programme made important efforts last year to increase participation in national events and improve communication and information dissemination. Further to these efforts, the Commission is revising the communication strategy and will complement the overall strategy by pursuing dedicated communication activities in specific areas.

6.2. Engagement of stakeholders and project management continuity

Members of the specialised working groups established by the ISA Committee should report nationally to the ISA Coordination Group members to ensure that ISA solutions are aligned with needs and initiatives at national level. In addition, relevant stakeholders from the respective public administrations should be identified by the ISA Coordination Group members and involved on an ad-hoc basis in formal meetings and workshops 11.

6.

The Commission proposes to support the Member States in implementing this recommendation by, for example:


7.

- making use of the established networks with EU regions


8.

- exploring means of remote participation in meetings (videoconferences)


- other means to be identified together with the Member States.

In addition the ISA programme will give priority to activities to assess the ICT implications of EU legislation, which Member States consider to be an important issue that has not yet been addressed12.

The ISA programme should ensure continuity in the project management of actions by analysing the issues leading to the high overall turnover of human resources in the programme and by identifying mitigation actions13.

The evaluators recalled the recommendation from the IDABC final evaluation that the Commission should ensure that the contractual frameworks are in place in due time for the launch of the next programme, allowing actions to be launched as soon as a work programme is adopted14.

The Commission strongly promotes mobility among its staff. In order to assure that mobility does not compromise project continuity, and to exploit competences available in different services (which cannot all be found in the same unit), the Commission is exploring synergies among its services by promoting inter-service contacts and stronger cooperation between services.

The Commission is closely following up the procedure for framework contracts, which will assure the timely launch of the required call for tenders.

6.3. Avoiding overlaps and duplications, increasing reusability and ensuring sustainability

The ISA programme should apply a business case approach in the selection of new actions proposed by Member States and Commission services and involve the Commission’s IT governance bodies in the evaluation of business cases for proposals from the Commission services15.

The Commission’s IT governance bodies should ensure that external synergies between ISA actions and other EU initiatives are identified and documented upfront.

DIGIT's participation in the IT governance bodies will ensure that the ISA programme is associated with the proceedings of these bodies so as to better identify cases and where ISA solutions can be reused16.

The ISA programme should establish a control mechanism to ensure the reuse of ISA solutions. Reusability should be a feature of the ISA solutions selected for funding. These solutions must include documentation for their reuse.

In this respect, new proposals should include an adequate governance structure to support reuse. Reusable solutions and opportunities for reuse should be identified and communicated to relevant stakeholders 17.

The ISA programme should consider in due time the sustainability of ISA solutions by identifying different sustainability options, such as charge-back methods, financial support from Commission services that have developed specific ISA solutions, and financial support from an ISA follow-on programme or other EU programme1819.

9.

The Commission takes a holistic approach to addressing the reusability and sustainability of interoperability solutions by acting at different levels:


1. At governance level, by improving the links between the Commission’s IT governance20, the ISA programme and through the ISA committee with the Member States. The ISA is currently presented annually to the CTI and individual ISA actions led by Commission services must be approved by the ISPMB;

2. At the strategic level, by putting more emphasis on reusability and sustainability in future ISA work plans and if necessary by a possible review of the European Interoperability Strategy21 (EIS) 22;

3. At operational level, by taking action to ensure better reusability of interoperability solutions and to develop sustainability enablers (e.g. European Interoperability Architecture (EIA), European Federated Interoperability Repository (EFIR), Assessment of trans-European networks supporting EU policies, Sharing and reuse strategy) 23 and by considering possible means of financial sustainability including the proposed CEF.

Finally, in line with the evaluator's recommendation, the ISA unit will identify actions producing concrete results by reviewing ISA solutions every two years as specified in Article 13(2) of the ISA Decision24.

7. Conclusions

In the on-going implementation of the ISA programme, the Commission will pay the utmost attention to the shortcomings highlighted and the associated recommendations, analysing them to validate and address the issues raised, wherever appropriate in close cooperation with the Member States.


1Decision No 922/2009/EC of the European Parliament and of the Council of 16 September 2009 on interoperability solutions for European public administrations (ISA), OJ L 260, 3.10.2009, p. 20.

2Decision 2004/387/EC of the European Parliament and of the Council of 21 April 2004 on interoperable delivery of pan-European eGovernment services to public administrations, businesses and citizens (IDABC), OJ L 144, 30.4.2004 (see OJ L 181, 18.5.2004, p. 25).

3The full evaluation report is available at: ec.europa.eu/isa/documents

4 DG CONNECT, DIGIT, MARKT, SG, TAXUD.

5Article 1(2).

6Article 1(1).

7The ISA Coordination Group is a working group established by the ISA Committee to address issues referred to it, assist the Commission in translating the European Interoperability Strategy into concrete actions and coordinate the alignment of national and ISA actions.

8 Recommendation 1.

9 Recommendation 5.

10 Recommendation 8.

11 Recommendation 7.

12 Recommendation 3.

13 Recommendation 4.

14 IDABC Recommendation 9.

15 Recommendation 2.

16 Recommendation 6.

17 Recommendation 9.

18 Recommendation 11.

19 Related IDABC final evaluation recommendations: (1) The Commission should have further open discussions with its stakeholders to agree on which of the efforts and budget under the new ISA programme should be allocated to operating IT infrastructures, developing new actions and promoting/reusing existing solutions. (2) Information on project sustainability and on financial and operational sustainability should be rendered more visible and be better explained to external stakeholders.

20 Commission IT Governance bodies are: Comité Technique Informatique (CTI), High Level Commitee on Information Technologies (HLCIT), Activity Based Management Steering Committee covering IT (ABM+IT) and the Information Systems Project Management Board (ISPMB).

21 Also addressing one recommendation in the IDABC final evaluation: a common ‘promotion’ document, focusing on the policy alignments and the synergies between the different eGovernment programmes, should be produced.

22 The "Annex I to the Commission communication on interoperability - European Interoperability Strategy (EIS)" is available at ec.europa.eu/isa/documents/isa_annex_i_eis_en.pdf

23 These actions can be found in the recent ISA Work Programme at: ec.europa.eu/isa/library/index_en under the 'ISA Work Programme' heading.

24 Recommendation 10.

EN EN