2nd independent External Evaluation of the European Union Agency for Fundamental Rights - Main contents

Document date	20-02-2018
Publication date	21-02-2018
Reference	5999/18
From	Commission Services
External link	original article
Original document in PDF

Council of the European Union Brussels, 20 February 2018 (OR. en)

5999/18

FREMP 9 JAI 87 COHOM 18 COPEN 27 SOC 42 ASIM 6 MIGR 12 DAPIX 31 ENFOPOL 56

COVER NOTE

From: Commission Services

To: Delegations

No. prev. doc.: 11263/13 REV1, 11263/13 REV1 ADD1, 11263/13 REV1 ADD2

Subject: 2nd independent External Evaluation of the European Union Agency for Fundamental Rights

Delegations will find enclosed a report submitted by COM on the 2nd independent External

Evaluation of the European Union Agency for Fundamental Rights.

Ref. Ares(2018)672541 - 05/02/2018

2nd independent External Evaluation of the

European Union Agency for Fundamental Rights

European Union Agency for Fundamental Rights

Final Report

31 October 2017

Berlin | London | Brussels | Los Angeles |New York | Washington, DC | Zurich 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Final report

Executive Summary .................................................................................................. 3

• 1. 

  Introduction ...................................................................................................... 9

Scope and objective of the evaluation .................................................................. 9

Overview of the evaluation questions ....................................................................... 9

Methodological Approach .................................................................................. 12

• 2. 

  The European Union Agency for Fundamental Rights ....................................... 15

Background to the FRA ...................................................................................... 15

Brief history of the FRA and mandate ..................................................................... 15

The FRA’s organisation and developments since 2012 ......................................... 16 Future of the FRA ............................................................................................... 30

• 3. 

  Evaluation context ........................................................................................... 34

The FRA’s intervention logic ............................................................................... 34 Policy context .................................................................................................... 42 Important issues ................................................................................................ 49

• 4. 

  Evaluation criteria ........................................................................................... 58

Relevance .......................................................................................................... 58 Effectiveness, Utility and Impact ........................................................................ 73 Added Value .................................................................................................... 102 Assessing Coordination and Coherence ............................................................ 111 Efficiency ......................................................................................................... 128

• 5. 

  Conclusions and recommendations ................................................................ 141

Conclusions ..................................................................................................... 141 Recommendations ........................................................................................... 144

Case study 1 – Violence Against Women survey .................................................... 148 Case Study 2 – FRA activities in hotspots ............................................................... 159 EU-MIDIS II – Muslim report social media plan report launch analysis ................... 172 Appendix 1 – Full intervention logic ...................................................................... 193 Appendix 2 – Survey results .................................................................................. 212 Appendix 3 – List of stakeholders consulted .......................................................... 213 FRA Impact – Twitter Sentiment analysis .............................................................. 226

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Executive Summary

The purpose of the evaluation, as set out in the Founding Regulation (EC/168/2007) of the Agency, was to provide external and independent review of the progress and achievements since the first

independent evaluation of the European Union Agency for Fundamental Rights (FRA) in 2012.

The European Union Agency for Fundamental Rights (FRA) is the EU’s centre for fundamental rights expertise. The Agency’s Founding Regulation tasks the Agency to ‘provide the relevant institutions, bodies, offices and agencies of the Community and its Member States when implementing Community law with assistance and expertise relating to fundamental rights in order to support them when they take measures or formulate courses of action within their respective spheres of competence to fully respect fundamental rights’. 1 The agency is hence a consultative independent expert body without

legislative or regulatory powers.

In line with its Founding Regulation and its mandate, the Agency strives to achieve its objectives by implementing projects and activities mainly within the thematic areas of a five-year Multi-Annual

Framework, which fall broadly under different chapters of the Charter of Fundamental Rights.

In order to contribute towards safeguarding and ensuring full respect for the fundamental rights across the Union, the FRA performs the following main tasks:

• 1. 

  Data collection and research – FRA collects and analyses data and information from all or several EU Member States and identifies current issues and challenges in the area of fundamental rights by providing a comparative perspective.

• 2. 

  Advice and expertise – FRA provides assistance and expert opinions on fundamental rights to its stakeholders and to its networks, at national, EU and international level.

• 3. 

  Disseminate information and raise awareness – FRA’s communication tasks focus on raising awareness to the public and to professions that are working on protecting fundamental rights.

Since 2016, the activities undertaken by the Agency have shifted and expanded in different ways. First of all, the number of ad-hoc legal opinions delivered by the Agency to the European Institutions increased to six in 2016. Second, the Agency started undertaking activities in which it had previously not taken part. 2016 saw, for the first time, agency staff being deployed to crisis zones to provide direct on the ground advice. The Agency took the decision to send members of staff to Greece in order to observe and provide advice and feedback at a very practical level on the way in which the hotspots functioned with respect to fundamental rights standards. The deployment of staff on a regular basis to Greece

continued in 2017.

Evaluation findings

Since its creation in 2007, the FRA has developed from an organisation purely in charge of collecting information to support, into one that is advising on Fundamental Rights in operational theatres such as

at migrant hotspots. This has been achieved through the realisation that the needs on the ground

1 Council Regulation (EC) Ni 168/2007 i of 15 February 2007 establishing a European Union Agency for Fundamental Rights (hereafter, the

Founding Regulation), Article 2.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

require various formats, even those of a very practical nature, in order to efficiently provide ‘assistance and expertise relating to fundamental rights’ as required by the Founding Regulation.

Relevance

In respect to the Agency’s objective as stated in its Founding Regulation, the work undertaken by the Agency is highly relevant to its stakeholders at the EU level, as well as at the national one. As the

pressure on Member States’ authorities, as duty bearers, increases in the field of asylum procedures and Returns Policy, for instance, the need to design fundamental rights safeguards and to create evidencebased advice on their implementation has increased the Agency’s relevance for duty bearers working in key areas. Furthermore, the Agency’s strategic and programming documents are flexible enough to adequately respond to any changes in the needs, as demonstrated by the Agency’s work in migrant hotspots.

Stakeholders’ perceptions of the relevance of the Agency depend to a large extent on how well they know the work of the Agency, and what they understand the role of the FRA to be. These perceptions vary from providing information and research to being the EU’s fundamental rights watchdog. There is not always a common understanding of what the objectives of the Agency are. This lack of common understanding can lead to situations where some stakeholders have a more negative view of the Agency’s impact given that they expect its objectives to be much wider than they actually are.

However, the majority of stakeholders mentioned that the mandate of the FRA should be amended to explicitly include judicial cooperation and police cooperation in criminal matters.

Effectiveness, utility and impact

Impacts are the highest level of effects and consequently the most difficult to measure and attribute to specific activities. One of the specificities of the Agency is its collection of data through primary and original research in order to inform policy-making. This dual aspect is key to the achievement of certain impacts. Given the mandate of the Agency to provide assistance and expertise, the impacts it achieves will depend on the readiness of EU and national policy-makers to act on these. A key aspect of the impact the Agency has is reflected in its dual role of undertaking research and analysis and using this to support and influence policy-making at the EU and national level.

The quality of the Agency’s research outputs is undisputed by stakeholders consulted. The Agency’s research is praised for its quality, and the fact that it provides data which is comparable across the EU and over time, as well as scientific and objective. The FRA has been recognised to be very efficient at using existing research in different forms of publication and dissemination. The creation of the Fundamental Rights Promotion Department (FRPD) partly stemmed from this urge to make better use of existing research, in addition to the FPRD’s task to strengthen fundamental rights awareness in Member States, and develop/foster relations with existing and new stakeholder communities. The Agency consistently and increasingly uses this data to inform the opinions it provides (such as the ad-hoc legal opinions requested by the European Parliament or one of the other two EU co-legislators (European Commission and Council).

Most of the impacts that the Agency has at national level are indirect, through cooperation with national partners or the use that national actors make of the FRA’s research. Some direct impacts were identified, in particular through country visits, or in the development of legislation.

At the EU level, the impact of the FRA is increasingly visible; for example in December 2016 the European Parliament adopted a resolution on the situation of fundamental rights in the European Union in 2015, which contains 20 references to the findings and work of FRA. The Violence Against Women survey undertaken by the Agency has been instrumental in the accession of the EU to the

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Istanbul Convention. Recent draft legislation (for instance the proposed European Travel Information and Authorisation System (ETIAS) Regulation) borrow extensively from the Agency’s legal opinions.

However, the public visibility of the Agency and the impact it can have are not always correlated. The importance of the unofficial links between the Agency and other EU Institutions, bodies, offices and agencies is extremely important in the impact the FRA has at EU level in supporting them to develop legislation while taking into account Fundamental Rights principles. These interactions are not visible to

the outside world but arguably have a large impact.

The FRA’s outputs are easily available and increasingly developed in a format which is more user friendly than in the past. However, the outputs, in particular large research reports, were not always seen to be as accessible as they could be, prompting some stakeholders to call on the Agency to think even more carefully about its target audience. At the same time, it can be noted that the Agency’s Violence Against Women survey main results report, which is just under 200 pages in length, has consistently resulted, year on year since its launch, in the highest number of downloads of any product by the Agency – more than the short ‘at a glance’ report or factsheet accompanying the survey, which were available in 22 languages. FRA has been active on social media such as Facebook, Twitter, YouTube and (to a lesser extent) LinkedIn. In all of these different platforms a steady increase of activities, followers/fans and likes has been noted. However more traditional ways of disseminating results such as traditional media and events are still important. The events organised by the Agency are very highly regarded by its

stakeholders in terms of content as well as innovation.

Added value

The FRA is contributing importantly, and in a unique way, to the promotion and protection of fundamental rights in the EU. FRA is the only organisation collecting and analysing data on fundamental rights at European level in a comparative and independent way. The dual role of the Agency in producing socio-legal analysis and providing evidence-based policy advice is key to its uniqueness

and conductive to the impacts it can have.

The FRA has an important role in providing a forum for discussions on fundamental rights involving relevant stakeholders from across the EU Member States. As outlined above, the evaluators conclude that added value can be found in FRA’s research activities, the conclusions and opinions, the stakeholder engagement, the independent nature of the Agency, as well as in its, so far rather indirect, monitoring

role.

Coordination and coherence

There is a robust structure in place for collaboration with relevant EU and national policy stakeholders, international organisations, and civil society. Furthermore, FRA has adequate informal channels in place which are used for exchange of information. There is an effective coordination and coherence particularly with other stakeholders involved in similar activities as FRA (e.g. the Council of Europe, OSCE, etc.). While a few overlaps have been highlighted, it seems as if complementarity between FRA and other actors is widespread. Furthermore, efforts are currently being made to improve cooperation, communication, and stakeholder engagement with the aim of making the cooperation structures in

place more effective.

In general, cooperation activities with the CoE and the UN are considered positive and effective according to most interviewees, and some have indicated that the cooperation has improved over time. Interesting stakeholder meetings as well as the FRA taking the initiative with other international

organisations were mentioned by interviewees as positive factors, in particular with the UN and the

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Council of Europe. Operational cooperation and joint publications such as the Handbooks which are produced in cooperation with the European Court of Human Rights were also highly regarded.

Efficiency

The Agency has achieved considerable efficiency in its operations and has implemented all the actions from the Common Approach (which defines a more coherent and efficient framework for the functioning of agencies) that it can in the current context. The Agency has been at the forefront of the implementation of Activity-Based Management, and the governance structure in place (in particular the existence of an Executive Board) is only now being implemented by other decentralised agencies. Some additional efficiency gains could be found, although this would require changes to the Founding

Regulation.

There seems to be a clear acknowledgement by internal stakeholders that the recent changes to the organisational structure were warranted and could offer both a streamlining of internal processes with the creation of the Corporate Services department and an improvement to the Agency’s communication

and dissemination activities through the creation of the Fundamental Rights Promotion Department.

There seems to be an acknowledgement that the Agency is at the limit of the human resources necessary to respond effectively to the increased demands placed on it with respect to data collection, having a presence on the ground in response to fundamental rights emergencies, and communication

and dissemination.

Recommendations

The following recommendations for actions are based on findings and conclusions of the evaluation and are divided under their evaluation criteria. The recommendations are addressed to the FRA, except

when expressly noted.

Overall

 The agency’s work is clearly highly regarded by stakeholders, the Agency should continue doing

what it does;

Relevance

 To the Commission, the Council and the Parliament: The Agency’s Founding Regulation should be

modified in order to bring the wording in line with the post-Lisbon reality and thus increase legal clarity. A revised wording of the regulation could stress the importance of the Charter as a now legally binding standard and make explicit that the regulation covers judicial cooperation in criminal

matters;

 The Agency should conduct further stakeholder consultation on the nature of the new activities

stakeholders reported requiring from the Agency, regardless of whether these activities are in the Agency’s current mandate, in order to more accurately capture the exact fundamental rights needs of the Agency’s stakeholders, which could potentially act as greater evidence for a revision of the

Agency’s mandate.

Effectiveness, Utility and Impact

Regarding meeting the needs of stakeholders:

 The FRA should continue to be responsive to requests (particularly on current emergencies like the

migration crisis) whilst not neglecting on-going research projects.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

 The need to plan research projects in advance should continue being balanced with ad-hoc needs

(e.g. allow re-scoping of long-term research projects to grant researchers the time to work on adhoc needs).

 FRA should focus on gaining more visibility at the national level. One way is to engage more closely

with those NGOs and NHRIs that could build a link to authorities.

 The Agency should continue to set clear and realistic targets both on thematic and strategic levels.

There could be a discussion on whether the scope of the thematic areas shall be limited (e.g. by combining one or more thematic areas in one) in order to limit the scope of FRA’s activities in light of resource limitations.

 On a methodological point, it would be useful to structure the mid-term strategic review differently

in order to more easily detect how FRA is doing in respect to each objective (e.g. providing a clear comparison between target and progress).

 The Commission could look into the relevance, appropriateness and need of the MAF that currently

defines the thematic areas for a period of five years and through a decision adopted not by the Agency but the Council of the European Union.

 To the Commission, the Council and the Parliament; the FRA should be permitted to initiate

research in the area of police and judicial cooperation in criminal matters, and social rights, in order to meet the needs of stakeholders at the national and EU level.

Regarding the Agency’s communication:

 The Agency should focus on increasing its impact at the national level by building country-specific

knowledge and expertise on Member States, in particular those that warrant specific attention at any point in time.

 FRA should focus on those stakeholders in its dissemination strategy that have a multiplying effect

(e.g. more interaction with media in addition to the work on social media which reaches mainly individuals already aware of FRA’s activities).

 FRA should clarify with its diverse set of stakeholders who its key audience is – namely, EU

Institutions and Member States’ governments – which would serve to explain its different levels of engagement with various stakeholders that include and go beyond these groups.

 The Agency should continue to capitalise on the wealth of research and analysis it has accumulated

by ‘re-packaging’ it in publications and other activities that could have a wider impact, especially at the national level.

Added value

The European Parliament and Council should ensure that the Agency can:

 Continue to focus on elements that make it distinct from other Fundamental Rights organisations

and add value to its stakeholders; in particular:

o the collection and analysis of comparable data;

o support to policy-makers in developing fundamental-rights-compliant legislation;

o real-time and practical assistance and advice based on support and activities undertaken in the field (such as the Agency’s work in migration hotspots);

o the Agency’s independence should continue to be guaranteed.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

 Provide further forums for discussions on fundamental rights involving relevant stakeholders from

across the EU Member States, as well as with additional stakeholder groups.

Coordination and Coherence

The Agency should:

 Continue the complementary collaboration with EU/national policy stakeholders, international

organisations, and civil society, and investigate the complementarities that were highlighted by a handful of stakeholders in the evaluation.

 Where possible within FRA’s remit, formalise greater cooperation between international

organisations with cooperation agreements as well as the production of more joint outputs.

 Continue expanding the level of cooperation with civil society.

Efficiency

The Agency should:

 Put the business case forward for further investment in communication and in dissemination

activities in order to maximise the impact of the Agency’s activities.

 Clearly delineate and communicate the roles and responsibilities of staff and the new departments

amongst the Agency’s staff members.

 Following and endorsing the recommendation from the Agency’s first external evaluation, create

the title of Deputy Director with the task of day-to-day management of the Agency, in order to prioritise the Director’s activities towards leadership, outreach and cooperation.

 Sufficient resources should be provided for research activities corresponding to new requests

received annually by EU Institutions for more research evidence on fundamental rights issues;

 The Agency should be given an increase in its human and financial resources, in order to enable the

Agency to reach its critical mass in terms of impact, which will allow the Agency to effectively respond to the increased demands placed on it with respect to data collection, having a presence on the ground in response to fundamental rights emergencies, and communication and

dissemination.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

• 1. 

  Introduction

  This document constitutes the final report for the 2nd independent External Evaluation of the European Union Agency for Fundamental Rights for the 2013–2017 period.

  Scope and objective of the evaluation

1.1.1. Objectives of the evaluation

The purpose of the evaluation, as set out in the Founding Regulation (EC/168/2007) of the Agency, is to provide external and independent review of the progress and achievements since the first independent evaluation of the FRA. The Management Board and the Commission have agreed that external evaluation will be undertaken at five-year intervals. As such this evaluation covers the Agency’s activity after its first evaluation in 2012, thereby evaluating the Agency’s activitie s from 2013 to 2017. In order

to review the Agency’s progress and achievements, and in line with the terms of reference, this

evaluation:

takes into account the tasks of the Agency, the working practices and impact of the Agency on the protection and promotion of fundamental rights; 

 assesses the possible need to modify the Agency’s tasks, scope, areas of activity or structure; 

includes an analysis of the synergy effects and the financial implications of any modification of the tasks; and 

takes into account the views of the stakeholders at both Union and national levels. 

The evaluation also assesses the FRA’s alignment with the Common Approach for Decentralised Agencies and its implementation of the Commission’s Roadmap. More specifically, it examines the

strengths and weaknesses of the FRA’s current operating model versus what has been envisaged in the Common Approach. Finally, the evaluation assesses the internal performance of the Agency.

Overview of the evaluation questions

The assessment of the overall impact of the FRA is based on the evaluation of the Agency and its working practices against the evaluation criteria. Below, we provide the rationale behind this concept.

First, it is necessary to evaluate the performance of the FRA against the evaluation criteria outlined in the Tender Specifications, namely: relevance, effectiveness, impact, utility, added value, coordination and coherence and efficiency.

Figure 1 shows the different evaluation criteria placed in the context of an EU intervention. The figure

below shows the different evaluation criteria placed in the context of the FRA’s intervention logic

developed in Section 2. Integrating the Performance Measurement Framework to the intervention logic ensures that the information and indicators collected through the Agency’s existing monitoring and evaluation framework can be used as extensively as possible in this evaluation. This, in turn, will allow for resources to be deployed to add value in other areas of the assignment, and in particular in assessing the Agency’s impacts.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 1: Evaluation criteria

Utility Added-value

External factors

Relevance

Impacts

Other EU Policies

Needs Coordination Effects

and Coherence Results

Objectives Inputs Activities Outputs

FRA Efficiency

Effectiveness

Evaluation criteria Items covered by the PMF

The overarching evaluation questions are presented in Table 1.

Table 1: Overview of evaluation questions

Overarching evaluation question

EQ1 To what extent is the Agency’s original objective still relevant to addressing the needs, problems and fundamental rights issues within the EU of the FRA target groups?

Relevance EQ2 To what extent are the needs of the relevant stakeholder groups met by the Agency’s mandate and actual performance?

EQ3 Have the recommendations on the relevance from the previous evaluation been implemented by the Agency

How successful has FRA been in achieving the expected effects (outputs, results, EQ4 impacts), in light of its objectives, mandate and tasks, as defined in its Founding

Regulation?

EQ5 To what extent do internal processes and ways of working impact on the Agency’s ability to perform its essential tasks?

EQ6 To what extent are the Agency’s activities effectively resourced?

Effectiven

ess and EQ7 To what extent are the Agency’s outputs and activities useful to its various

Utility stakeholders?

EQ8 To what extent did FRA activities have an impact on EU policy and practices in MS/third countries/NHRBs/fundamental rights field?

EQ9 To what extent have the objectives set out in the multi-annual and annual work programmes for the years 2013 to 2017 been accomplished?

EQ10 To what extent does the impact achieved by the FRA’s activities correspond to and

meet existing stakeholders’ needs?

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Overarching evaluation question

EQ11 Are there any additional outputs/results that were not foreseen initially in the multiannual and annual work programmes?

How should FRA activities be adapted in order to more effectively respond to its EQ12 multi-annual and annual priorities and the potential requests for services and

activities from EU Institutions/MS/other stakeholders and international bodies?

EQ13 To what extent are FRA’s outputs fully accessible and made use of by relevant stakeholders? To what extent does the Agency effectively communicate on its activities?

EQ14 To what extent have the recommendations from the 2012 evaluation related to the effectiveness and impacts been implemented?

EQ15 What is the overall added value of the FRA?

EQ16 To what extent has the scientific quality of the Agency’s outputs and activities been ensured?

Added

value EQ17 What has been the unique contribution of the Agency to the promotion and

protection of fundamental rights in the EU?

EQ18 To what extent have the effects of the Agency’s activities been achieved at lower costs because of the Agency’s intervention?

EQ19 To what extent is the Agency acting in close cooperation with the CoE and UN to avoid duplication and in order to ensure complementarity?

EQ20 To what extent is the Agency ensuring appropriate coordination with relevant stakeholders to foster synergies and avoid duplication?

Coordinati EQ21 To what extent are the procedures to ensure this coordination and cooperation on and effective to ensure the Agency’s activities are coherent with the policies and activities

Coherence of its stakeholders?

EQ22 To what extent are the FRA’s objectives and activities coherent with 1) the Common Approach of the European Parliament, the Council and the European Commission and 2) the European Commission’s Roadmap for raising the effectiveness and improving governance of the decentralised agencies

EQ23 Does the Agency’s organisational and budgetary structure contribute to effectiveness and efficiency of its operations?

EQ24 Is the size of the budget and human resources appropriate and proportional to what FRA is expected to achieve? Is it sufficient for reaching a critical mass of impact?

Efficiency EQ25 Is there a good balance between administrative and operational budget?

EQ26 To what extent has the Agency been successful in creating synergies and an optimal use of combined resources allocated for the implementation of its mandate and tasks to manage the operation?

EQ27 Is the Agency reporting in accordance with Commission guidance and templates?

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Methodological Approach

The third phase of the project, the final reporting, focused on the analysis of the evaluation findings in order to answer the evaluation questions, while ensuring that the seven evaluation objectives are

addressed.

In addition to the desk research, three online surveys as well as an in-depth interview programme were used for the data collection. The methods were chosen to target a wide range of stakeholder groups and to address the evaluation objectives. The culmination of these tasks is the submission of this draft final report.

This draft final report presents the analysis of all the results and data collected to date. The analysis presented in section 4 should be understood as the final analysis.

Data collection

Large-scale interview programme

Alongside the collection of data through three online surveys, and building on the familiarisation interviews conducted in Phase 1 and 2, the contractor planned to conduct 60–70 interviews during the

large-scale interview programme. To date, the contractor’s Research and Analysis Team has successfully met this target, having conducted 107 interviews with key stakeholders at the international, EU and the national level, as illustrated in Table 2. A full list of people interviewed can be found in Appendix 3.

Table 2: Interviews undertaken

Stakeholder category Number of interviews

FRA Staff 26 National level (NLOs, civil society 27 organisations) EU bodies and Institutions 24 International level 8 FRP/FRA Advisory Panel members 5 FRA Executive Board/ Management Board/ 17 Scientific Committee Total 107

Initial difficulties which the contractor encountered in setting up interviews, including a limited response from EU Institutions and bodies, were mitigated through the inclusion of further potential contacts as supplied by the Agency.

Online surveys

The contractor developed three online surveys:

one internal survey targeted at FRA staff; and 

two external surveys targeted at: 

o external stakeholders from the EU Institutions and bodies,

o other (non-EU institutional) external stakeholders.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Respondents were consulted on a number of issues dedicated to their specific stakeholder groups as well as more general questions on the Agency’s impact, relevance and utility. The surveys were discussed and tested with FRA staff and ultimately launched on 22 May, with a deadline set for 16 June and two reminders were sent over the period. The deadline for the survey was extended to 18 August, and two further reminders, including from the FRA’s Director, were sent over the period. These surveys

received a combined 156 responses from the Agency’s stakeholders, as listed in Table 3.

Table 3: Survey responses

Survey Responses

Internal stakeholder survey 66 responses

External EU institutional stakeholders 14 responses survey

External non-EU institutional stakeholders 76 responses survey

Total 156 responses

After concern was raised within the Agency about the anonymity of respondents, the research team stressed the anonymity of the responses and that the Agency will not be able to view the individual responses. These mitigation measures significantly improved the internal stakeholders’ survey response

rate in the final weeks of the survey.

Given the current low response rate for the external surveys, the research and analysis team undertook mitigation measures, including the extension of the interview programme to cover stakeholders underrepresented in the survey responses and increasing the interview programme from between 60

and 70 interviews to over 100 interviews.

Additionally, as discussed above, an extension on the availability of the online survey was used to successfully capture as many respondents as possible, alongside a fourth reminder email, targeted specifically to those that had not currently completed the survey, highlighting the reduced engagement we have seen from the target groups. This extension significantly increased the engagement from other

(non-EU institutional) external stakeholders.

Desk research

The contractor’s research and analysis team continued to review the relevant policy and programmatic documents as well as any additional documents provided by the Steering Group and interviewees. A bibliography was established to track all data sources consulted, which will provide a record of secondary qualitative and quantitative data sources that can be made available to members of the team. The documents have also informed the development of the coding framework and were

accordingly partially uploaded to NVivo 2 in order to ensure that all documents were included in the

analysis.

2 NVivo is a Qualitative Data Management software in which data can be classified, sorted and arranged in order to examine relationships and

combine analysis with linking, shaping, searching and modelling.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Impact case studies and sentiment analysis

Two Impact case studies on the Violence Against Women survey and the Agency’s activities on the ground at hotspot locations in Greece have been undertaken in order to delve more deeply into how the

Agency achieves impact.

Furthermore, a sentiment analysis of the tweets surrounding the Fundamental Rights reports’ release in 2017 and in response to the Fundamental Rights Forum in 2016, has been undertaken as well as an analysis of the Agency’s social media plan for the release of the publication of the ‘Second European

Union Minorities and Discrimination Survey (EU-MIDIS II): Muslims – Selected findings’ report.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

• 2. 

  The European Union Agency for Fundamental

Rights

Background to the FRA

The European Union Agency for Fundamental Rights (FRA) is the EU’s centre for fundamental rights

expertise. The Agency’s Founding Regulation tasks the Agency to ‘provide the relevant institutions,

bodies, offices and agencies of the Community and its Member States when implementing Community law with assistance and expertise relating to fundamental rights in order to support them when they take measures or formulate courses of action within their respective spheres of competence to fully respect fundamental rights’. 3 The agency is hence a consultative independent expert body without

legislative or regulatory powers.

Brief history of the FRA and mandate

The FRA was established building upon the independent European Union Monitoring Centre on Racism and Xenophobia (EUMC) which had been established in 1998. 4 Shortly after the creation of the EUMC,

work started on the development of a Charter aimed at consolidating ‘the fundamental rights applicable

5

at Union level’. Work on the Charter culminated in the proclamation by the Council, the Commission and the Parliament of the Charter of Fundamental Rights of the European Union (the EU Charter) in 2000. The Charter only became legally binding with the entry into force of the Lisbon Treaty in December 2009. As the Agency was created in 2007, its mandate does not prominently refer to the Charter as a core standard. Moreover, the regulation establishing the Agency still uses the term

‘European Community’ and not ‘European Union’, despite the fact that on the entry into force of the

Treaty of Lisbon, the European Community ceased to exist and was replaced by the European Union. This has contributed to a lack of legal clarity as to whether the Agency is mandated to deal also with

issues outside the former so-called first ‘Community’ pillar. 6

Even after the Treaty of Lisbon entered into force, the Multi-Annual Framework as adopted by the Council of the European Union and defining the thematic areas covered by the Agency did not extend to what is often referred to as the ‘former third pillar’; t hat is, police and judicial cooperation in criminal

matters. This despite the fact that the Founding Regulation itself stated that ‘nothing in this Regulation

should be interpreted in such a way as to prejudice the question of whether the remit of the Agency may be extended to cover the areas of police cooperation and judicial cooperation in criminal matters’. 7 The agency’s Management Board, responding to a consultation of its stakeholders regarding its activity

3 Council Regulation (EC) Ni 168/2007 i of 15 February 2007 establishing a European Union Agency for Fundamental Rights (hereafter, the

Founding Regulation), Article 2.

4 Council Regulation (EC) No 1035/97 i of 2 June 1997 establishing a European Monitoring Centre on Racism and Xenophobia.

5 1999 Council Conclusions (Cologne)

6 The Treaty of Maastricht (Treaty on European Union), creating the European Union established the ‘three pillar structure’ of the EU; the first

‘Community’ pillar comprised the European Community, the European Coal and Steel Community and EURATOM, following the supra-national model, the other two, covering Common Foreign and Security Policies (CFSP) and Police and Judicial Co-operation in Criminal Matters (PJCCM) were respectively known as the second and third pillars; they were both more strongly skewed towards in inter-governmental cooperation model.

7 Consideration number 32 of the founding regulation.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

for the period 2018–2022 8 , has suggested that the Multi-Annual Framework cover the former third

9

pillar .

organisation and developments since 2012

The FRA’s

2.2.1. Organisation

A key aspect of the FRA’s activities relates to the organisational structure and governance of the Agency, and thus one of the key elements to be assessed by this evaluation are the strategic choices

made by the Agency. After having been relatively stable over the period under evaluation, the structure of the Agency was amended twice in 2016 following decisions from the Director. The first change took place in April 2016 with the renaming and reorganisation of the Fundamental Rights Promotion

Department. Following the change, the department’s focus expanded. It was previously responsible for

the management of a number of external stakeholder groups – including National Liaison Officers representing Member States, as well as Equality Bodies and National Human Rights Institutions. This has now been expanded to include the management of media and social media. In December 2016, the Administration and the Human Resources and Legal Services departments were joined to form the Corporate Services department. Finally, Senior Policy Managers were appointed for each of the three

operational departments (Equality and Citizens’ Rights and Freedoms and Justice) as well as for the

Fundamental Rights Promotion Department. The role of the Senior Policy Managers is to be a horizontal one and non-project-specific. The structure of the FRA is presented in Figure 2 and the key elements are explained below.

Figure 2: The FRA’s organisational structure

Management Board

Scientific

Director Executive Board

Policy

Office Advisor

Corporate Services Equality and Freedoms and Fundamental

Department Rights Citizens’ Department Justice Department Rights Promotion Department

Source: http://fra.europa.eu/en/about-fra/structure

8 Consultation of stakeholders for the Multi-Annual Framework 2018–2022, 2015

9 Opinion of the Management Board of the EU Agency for Fundamental Rights on a new Multi-Annual Framework (2018–2022) for the agency,

Vienna, 12 February 2016

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

The FRA has four bodies: a Management Board, an Executive Board, a Scientific Committee and the Director. In terms of departments there are, next to the Director’s Office, four departments. The following table outlines in more detail their composition and activities. The Agency ‘shall fulfil its tasks in complete independence’. 10 This has also implications for the composition of the Agency’s structure. Importantly (and different from other EU agencies), the Managing Board is composed of one independent person appointed by each Member State ‘having high level responsibilities in an independent national human rights institution or other public or private sector organisation’. 11

Table 4: Overview of the FRA’s governance bodies, their composition and activities

Governance Composition Activities body

Management  One independent human rights  ‘The Management Board shall ensure that the Agency

Board expert from each of the 28 performs the tasks entrusted to it. It shall be the

Member States. Agency’s planning and monitoring body.’ 12

 One independent person  As the Agency’s planning and monitoring body, the

appointed by the Council of Management Board (MB) is the main decision-maker in Europe. the process of strategic planning.

 Two representatives of the  The Management Board appoints the Director and the

European Commission. Scientific Committee.

Executive  Chairperson of the Management  It prepares the decisions of the Management Board,

Board Board further assists and advises the Director.

 Vice Chairperson of the

Management Board

 Two further members of the

Management Board

 One of the representatives of

the European Commission

Director and -  The Director is responsible for implementing the tasks of

Director’s the FRA and for its staffing.

office  The Director is supported by the Directorate which

guides and manages FRA. It ensures that FRA’s objectives

are met in an efficient and effective way and in

accordance with the direction given by the Management

Board.

Scientific  Eleven independent human  ‘In order to ensure the high scientific quality of the work

Committee rights experts. It is selected of the Agency, the Agency should avail itself of a through an open call for Scientific Committee in order to guide its work by means applications of scientific objectivity.’ 13

 It guarantees the scientific quality of the FRA’s work.

10 Art6. 16 para 1 of the founding regulation.

11 Art. 12 para 1 of the founding regulation.

12 Council Regulation (EC) No 168/2007 i of 15 February 2007 establishing a European Union Agency for Fundamental Rights

13 Council Regulation (EC) No 168/2007 i of 15 February 2007 establishing a European Union Agency for Fundamental Rights

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

The FRA departments group thematic and operational teams. Specifically, they are the:

 Freedoms and Justice Department, which focuses on access to justice; asylum, migration and

borders; and information society, privacy and data protection. The department also includes the Agency’s Statistics & Surveys Sector.

 Equality and Citizens’ Rights Department, which focuses on equality and non-discrimination;

racism, xenophobia and related intolerance; Roma and migrant integration; and the rights of the

child. The department includes the Agency’s FRANET and Annual Report Sector.

 Fundamental Rights Promotion Department, which supports FRA in its role of providing evidencebased

advice through consultation, cooperation, awareness raising and communication activities.

 Corporate Services Department, which supports the operational work of the Agency and ensures

the highest standards for the management and development of the Agency’s human resources and the provision of legal services.

Director

The first Director of the Fundamental Rights Agency, Morten Kjaerum, left the Agency in 2015 after seven years of service and was succeeded by Michael O’Flaherty. The 2014 reform of the Staff

Regulations 14 introduced several changes to the structure of the Agency.

FRANET

FRANET is the Agency’s research network which consists of contracted organisations in all Member

States providing data and information on fundamental rights issues upon request. 15

Stakeholders

The work of the FRA is used by various stakeholders including, but not limited to, the institutions and bodies of the European Union, the Council of Europe, the United Nations, the Organisation for Security

and Cooperation in Europe (OSCE) and other international organisations.

The FRA maintains close links with relevant institutions and organisations at all levels:

 the European institutions, bodies, agencies and offices;

 national government authorities, particularly through its network of National Liaison Officers, who

act as main contact points for FRA in Member States, and through thematic working groups on Roma and hate crime bringing together Member States’ representatives to develop good practices

for implementation nationally;

 international organisations, such as the Council of Europe, the United Nations and the Organization

for Security and Co-operation in Europe (OSCE);

 civil society organisations and academic institutions;

 equality bodies and national human rights institutions (NHRIs);

14 Regulation (EU, EURATOM) No 1023/2013 of the European Parliament and of the Council of 22 October 2013 amending the Staff Regulations

of Officials of the European Union and the Conditions of Employment of Other Servants of the European Union

15 http://fra.europa.eu/en/research/franet

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

 the general public; 16

 National Parliamentary Focal Points.

2.2.2. Activities and tasks

In line with its Founding Regulation and its mandate, the Agency strives to achieve its objectives by implementing projects and activities mainly within the thematic areas of a five-year Multi-Annual

Framework, which fall broadly under different chapters of the Charter of Fundamental Rights.

In order to contribute towards safeguarding and ensuring full respect for the fundamental rights across the Union, the FRA performs the following main tasks:

• 1. 

  Data collection and research – FRA collects and analyses data and information from all or several EU Member States and identifies current issues and challenges in the area of fundamental rights by providing a comparative perspective. FRA is one of the few EU Agencies that generates its own data in areas where there is a lack of data at EU and Member State level. This is done through large-scale surveys as well as qualitative fieldwork. FRA’s research covers specific fundamental rights that are agreed in its Multi-annual work programmes and frameworks. FRA also collects data on the protection of fundamental rights and conducts legal analyses.

• 2. 

  Advice and expertise – FRA provides assistance and expert opinions on fundamental rights to its stakeholders and to its networks, at national, EU and international level. FRA develops its stand-alone legal or socio-legal published ‘Opinions’ on the basis of requests received from key institutions; namely, the European Parliament, Council or Commission. FRA can also issue opinions and conclusions to EU Institutions and Member States when they develop and implement EU legislation. Typically, FRA’s expert assistance is delivered through reports on an on-going basis, opinions delivered by request from an EU Institution, or, training materials and programmes and, more generally, by sharing ‘promising practices’.

• 3. 

  Disseminate information and raise awareness – FRA’s communication tasks focus on raising awareness among the public and to professions that are working on protecting fundamental rights. To this end FRA uses written material, online resources, social media, etc. It also provides online tools (Charterpedia) that can support legal practitioners in applying the charter in their work.

Since 2016, the activities undertaken by the Agency have shifted and expanded in different ways.

First of all, and covering activities traditionally undertaken by the Agency, the number of ad-hoc legal opinions delivered by the Agency to the European institutions increased to six in 2016 (as opposed to only one in both 2014 and 2015). Second, the Agency started undertaking activities in which it had previously not taken part. While Agency staff have traditionally been on the ground as part of their research and data gathering activities, 2016 saw, for the first time, agency staff being deployed to crisis

zones to provide direct on-the-ground advice. Following the EU–Turkey Statement of 18 March 2016, 17

and the decision to return irregular migrants to Turkey, the Agency took the decision to send members of staff to Greece, on a rotating basis, over a period of six months in order to observe and prove advice

16 http://fra.europa.eu/sites/default/files/fra_uploads/2016-fra-factsheet_en.pdf

17 http://europa.eu/rapid/press-release_MEMO-16-963_en.htm

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

and feedback at a very practical level on the way in which the agreement was implemented. The deployment of staff on a regular basis to Greece continued in 2017.

These activities are very resource-intensive and, while a level of flexibility is built in the Agency’s planning, have put a strain on the Agency in particular sectors (as discussed in greater detail in section 4.5 on efficiency).

Planning of work through stakeholder engagement and cross-thematic approach

With regard to the planned projects, and in order to implement its long-term strategic approach, the Agency has adopted a multi-annual planning method. According to its methodology, the Annual Work Programmes of FRA are prepared two or three years in advance through systematic dialogues and consultations with its stakeholders (also referred to as the N-2 approach). Following this method, the Agency can develop large multi-annual projects based on fieldwork research. In addition, this long-term planning approach allows for more robust forecasts in workforce and financial planning while maintaining its focus and acting on its mandate. This high-level overview enables the Agency to cover a

broad spectrum of fundamental rights issues and emergencies.

In practice FRA designs and plans its projects based on an ‘integrated cross-departmental approach’. Specifically, the FRA departments of Equality and Citizens’ Rights and Freedoms and Justice collaborate and engage with the project stakeholders and conduct relevant research and opinion development. The Fundamental Rights Promotion Department performs the communication and awareness raising

activities while engaging with key agency partners.

In addition, to ensure a flexible way of working, FRA categorises its planned projects in different priority levels:

 First Priority: Projects that follow up on past work, correspond to key EU priorities and are

considered essential to complete work in a specific area

 Second Priority: Projects which, although essential, could be postponed to next year owing to, for

example, unforeseen requests by stakeholders

 Third Priority: Projects that can be done only if funds become available in the course of the financial

year.

Projects

The FRA undertakes projects in all the thematic areas defined in its Multi-Annual Framework. In addition, the Agency delivers operational projects categorised as cross-cutting projects or activities covering all thematic areas, Bodies of the Agency and Consultation Mechanisms, and Operational Reserves. A table with examples of projects presented the Agency’s Annual Work Programme for 2016 can be found in Appendix 2. Each project has objectives which feed into the operational objectives

mentioned above and indicators which may help assess their effects.

FRA evaluation system

In order to demonstrate its performance, achievements and impact and to meet the requirements of its Founding Regulation, the Roadmap on the follow-up to the Common Approach on EU decentralised agencies, and the Implementing Rules to the Financial Rules of the European Union Agency for Fundamental Rights, the Agency has drafted and developed an Evaluation Policy, an Evaluation Guide and an Evaluation Plan. The Evaluation Policy sets out the principles, processes and tools that FRA uses

to conduct evaluation activities. Complementing its Policy, the FRA drafted the Evaluation Guide which

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

provides analytical guidance for evaluation activities and provides a detailed explanation of the responsibilities, project selection process and guidance on external evaluations. The FRA prepares every year an annual Evaluation Plan, based on the Evaluation Policy and Guide, which indicates the projects for which ex-ante or ex-post evaluation will be conducted.

In order to ensure that its activities meet the planned targets, and as part of its efforts to enhance its performance and implementation quality, FRA has employed a set of instruments:

 Programming Document – Following the adoption of the Common Approach on decentralised

agencies by the European Parliament, the Council and the Commission in 2012 and the entry into force of the Framework Financial Regulation (FFR), FRA introduced in 2016 a new planning instrument, in accordance with the EC guidelines, that combines Annual Work Programmes within a multi-annual planning on strategy and thematic level.

 FRA Project Planning Evaluation (FRAPPE) – At the project level, FRAPPE is a process that ensures

that before the commencement of a project relevant and concrete performance indicators are in place to assess the projects results, outputs and impacts. FRAPPE has now been embedded in the context of the Agency’s new integrated programming approach.

 MATRIX – MATRIX is the tool that FRA uses for its project management. The enhancement of

MATRIX, from an administrative tool to a planning and reporting one, was a recommendation of the first external evaluation of the Agency, endorsed by the FRA Management Board.

 Performance Measurement Framework (PMF) – The PMF is a system of key performance

indicators that contains indicators across levels (project, thematic, core, operational, etc.). Performance indicators are used to plan, monitor, evaluate and report FRA’s activities. As a result, they appear in both FRA’s Annual Activity Reports and Annual Work Framework (actual and planned, respectively). In addition, the PMF links to FRA’s Multi-Annual Framework as well as its Strategic Plan.

 Management Board Working Group ‘STRATEGY’ – STRATEGY aims to review and refine the

Strategic Plan objectives.

 Stock-Taking reports – these reports offer an overview of the main achievements and preliminary

impact analyses for each strategic priority as laid out by FRA’s Strategic Plan.

 Other consultation – to ensure stakeholders’ engagement in its work decisions, FRA involves its

stakeholders in its decision processes by issues consultation procedures.

2.2.3. Resources

Financial inputs

The Agency is financially supported mainly by European Union funds which are supplemented by the Austrian government’s contribution to the rent of the Agency’s premises and contributions from candidate and potential candidate countries that participate as observers. The Agency prepares two budgets annually: the draft statement of revenue and expenditure for year N + 1 as well as the draft estimate of revenue and expenditure for N + 2, to facilitate its multi-annual planning. Each year the Agency must present its planned budgetary expenditure as part of its Programming Document. The Annual Work Programmes are prepared and assessed two or three years in advance and agreed within FRA’s stakeholder network. The Agency must also publish its actual annual expenditure through the

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Consolidated Annual Activity Reports, in a retrospective way linking to its planned activities and in the

report on budgetary and financial management. 18

The Agency presents its planned and actual budget in two ways: in a traditional representation and in an activity-based budgeting (ABB) methodology. These are presented below. The following tables present the budget broken down by title and ABB activity of the FRA as presented in the FRA Annual Accounts and Annual Activity Reports. The Agency’s planned and actual expenditure has been stable around EUR

21 million since 2013. 19

18 http://fra.europa.eu/en/about-fra/finance-and-budget

19 Annual Work Programmes 2013 – 2016 and Annual Activity Report 2013-2015,

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Table 5: Annual actual expenditure broken down in titles for the years 2013–2016 and planned expenditure for the year 2017

2013 20 2014 2015 2016 2017 Actualised Actualised Actualised Planned Planned

Title 1 – Staff expenditure €11,091,131.55 €11,235,881.19 €11,176,568.08 €11,799,000 €13,069,500 Payments €10,949,608.58 €11,150,975.59 €11,071,193.63

Appropriations carried over €141,522.97 €84,905.60 €105,374.45 Title 2 – Buildings, equipment and miscellaneous operating

expenditure €2,636,810.61 €2,472,866.37 €2,526,187.04 €2,476,000 €2,136,000

Payments €1,813,381.51 €1,921,400.61 €1,953,283.54 Appropriations carried over €823,429.10 €551,465.76 €572,903.50

Title 1 + 2 – Support activities €13,727,942.16 €13,708,747.56 €13,702,755.12 €14,275,000 €15,205,500

Title 3 – Operational expenditure €8,197,795.20 €8,064,154.68 €8,182,094.30 €7,328,000 €7,499,000 Payments €2,516,449.40 €1,936,121.33 €2,458,812.64

Appropriations carried over €5,681,345.80 €6,128,033.35 €5,723,281.66

Table 6: Financial resources by ABB activity for the years 2013–2017. Please note that in 2013 the thematic area of ‘Roma Integration’ was not an independent area. In addition, the thematic area of Visa and border control was incorporated in the general Immigration area in 2014 and 2015

2013 21 2014 2015 2016 2017 (Planned) (Planned)

Financial resources by ABB activity

Asylum, immigration and integration of migrants €2,190,394 N/A N/A N/A €1,520,000

Information society €786,161 €621,459 €672,467 €830,421 €448,784 Visa and border control €66,107 N/A N/A N/A N/A Immigration and integration of migrants, N/A €2,368,484 €2,645,141 €2,908,474 €2,656,128

20 Annual Accounts of the FRA for the Financial year ended 31 December 2013

21 Annual Accounts of the FRA for the Financial year ended 31 December 2013

23 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

2013 21 2014 2015 2016 2017 (Planned) (Planned)

visa and border control and asylum

Racism, xenophobia and related intolerance €681,613 €206,575 €541,397 €280,056 €1,483,568 Roma integration N/A €1,262,434 €1,399,383 €509,300 €2,749,393 Discrimination €2,719,173 €3,937,197 €2,810,322 €2,395,967 €1,420,922 Rights of the child, including the protection of

children €1,393,240 €1,025,451 €1,430,729 €488,445 €862,997

Access to efficient and independent justice €1,667,761 €1,206,287 €1,639,388 €2,178,685 €753,767

Victims of crime, including compensation to victims

of crime €1,427 €255,292 €73,380 €763,450 €303,925

Cross-cutting activities N/A €1,235,730 €911,519 €1,502,580 €7,073,735

Research and data collection €2,500,823 €930,302 €795,880 €1,072,955 Included in

Communication and awareness raising €3,226,433 €3,112,347 €3,461,981 €4,379,731 cross-cutting Bodies of the Agency € 452,536 N/A N/A N/A activities Consultation mechanisms € 467,291 € 401,538 € 400,404 €342,523 expenditure

Operational reserves N/A N/A N/A N/A €155,000

Total operational expenditure €16,152,960 €16,563,096 €16,781,993 €17,652,586 €7,499,000 Total support expenditure €5,493,777 €4,954,987 €4,447,007 €3,950,414 €4,823,783 Total €21,646,737 €21,518,083 €21,229,000 €21,603,000 €22,732,000

24 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

The breakdown of the actual costs in the different thematic areas helps identify the activities on which the Agency has spent the most funding. The most active areas are Immigration and integration of migrants, followed by the area of Discrimination, which have seen an increase in funding of approximately 33% and a decrease of approximately 12% between 2013 and 2016 respectively. Access

to efficient and independent justice comes third.

The Activity-Based Budgeting (ABB) and Activity-Based Costing (ABC) started to be developed in 2007 and were the subject of a framework concept paper by the FRA in 2013, following recommendations from the Internal Audit Service (IAS) and in line with the Common Approach. This was an effort towards a wider Activity-Based Management (ABM) process, whereby indirect costs (e.g. salaries, rent, electricity, telecommunications) are allocated to the operational areas of activities. This format provides better understanding of the Agency’s activities and make the analysis and interpretation easier as it links

finances and thematic areas together.

As shown in Table 7 FRA’s budget over the years is considerably smaller than those of EASO and Eurojust which are – like FRA – JHA agencies. While, on average, the budgets of JHA agencies more than quadrupled between 2007 and 2017 FRA’s share of total revenue of all JHA agencies is declining: from

8.8% of total in 2007 to merely 3.0% in 2017.

Table 7: Comparison between FRA 22 , EASO 23 , EMCDDA 24 and Eurojust 25 total, staff, operational and operating

budget

2013 2014 2015 2016 2017 Total Budget

FRA 21,348,510 21,229,000 21,229,000 21,359,000 22,704,500 EASO 12,000,000 14,656,000 14,991,360 19,438,600 69,206,000 EMCDDA 16,057,482 15,183,962 15,333,962 15,393,962 15,807,164 Eurojust 32,358,660 32,449,671 32,994,000 43,539,737 47,879,237

Staff Budget FRA 11,513,000 12,185,000 11,961,000 11,799,000 13,069,500 EASO 5,044,000 6,143,000 6,176,360 9,126,500 16,521,014 EMCDDA 9,537,228 9,336,638 9,196,364 9,321,198 10,128,023 Eurojust 17,333,166 16,964,992 17,801,275 18,864,895 18,501,261

Operational Budget FRA 7,628,510 6,800,000 7,050,000 7,328,000 7,499,000 EASO 5,000,000 6,027,000 6,178,000 6,609,000 43,804,986 EMCDDA 4,322,017 3,777,495 3,887,623 4,348,475 4,452,593 Eurojust 7,751,294 8,056,979 8,399,464 8,196,156 8,276,674

Support/Operating Budget FRA 2,207,000 1,096,000 2,218,000 2,232,000 2,136,000 EASO 1,956,000 2,486,000 2,637,000 3,703,100 8,879,999 EMCDDA 2,198,238 2,069,828 2,249,974 1,724,288 1,226,547 Eurojust 7,274,200 7,427,700 7,617,612 16,478,686 21,601,302

22 All annual budgets available at: http://fra.europa.eu/en/about-fra/finance-and-budget/financial-documents

23 All annual budgets available at: https://www.easo.europa.eu/budget-finance-and-accounting

24 All annual budgets available at: http://www.emcdda.europa.eu/publications-seriestype/budget

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Human Resources

The Agency’s staff fall into two categories: statutory staff, including temporary agents (TA) and contract agents (CA), and non-statutory staff containing Seconded National Experts (SNE). Table 8 presents the planned staff divided by categories for the years 2013–2017 as provided by the Annual Work Programmes published by FRA each year. In addition to these categories of staff, the Agency has an extensive internship programme, with around 35 full-time trainees employed at any one time.

Table 8: Number of staff planned for each post (TA, CA, SNE) and the total planned Human Resources figure

Number of staff by year

2013 26 2014 27 2015 28 2016 29 2017 30

Posts planned HR planned HR planned HR planned HR planned HR Temporary Agents 78 75.0 75.0 74.0 72.0

Contract Agents 27.0 29.0 31.0 33.0 37.0

Seconded National Experts 10.0 9.0 9.0 9.0 10.0

Total Operational Human 115.0 113.0 115.0 116.0 119.0 Resources

By comparing the number of planned staff across the period 2013–2017 we can observe an increase in the number of the Contract Agents (from 27 in 2013 to 37 on 2017) while the number of Temporary agents decreased overall by six and Seconded National Experts remained at a similar level. Finally, it is obvious that the majority of the Agency’s human resource consists of Temporary Agents.

Staff allocation and actual allocation can also be broken down into the various ABB activity areas of the

Agency 31 it delivers. Table 9 presents this analysis using the publicly available information shown in the

FRA Annual Activity Reports for each year.

Table 9: Total annual actual human resources per title and ABB activity for the period 2013–2015

Total Annual Actual Human resources per title

2013 32 2014 33 2015 34 2016 35 2017 36 Thematic Areas

Asylum, immigration and integration of migrants 5.50 6.90 7.80 12.50 8.95 Information society and, in particular, respect for private life 7.60 3.90 3.20 6.84 2.85

and protection of personal data Visa and border control 0.00

26 Annual Work Programme 2013, FRA

27 Annual Work Programme 2014, FRA, December 2013

28 Annual Work Programme 2015, FRA, December 2014

29 Annual Work Programme 2016, FRA, December 2015

30 FRA Programming Document 2017-2019

31 The support activities include: Human Resources and Planning, Directorate and Administration activities

32 Annual Work Programme 2013, FRA

33 Annual Work Programme 2014, FRA, December 2013

34 Annual Work Programme 2015, FRA, December 2014

35 Annual Work Programme 2016, FRA, December 2015

36 FRA Programming Document 2017-2019

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Total Annual Actual Human resources per title

2013 32 2014 33 2015 34 2016 35 2017 36 Racism, xenophobia and related intolerance 1.40 1.80 3.90 2.58 5.70 Roma integration 5.90 7.10 6.42 8.70 Discrimination 15.30 10.60 8.10 10.06 8.20 Rights of the child, including the protection of children 2.30 5.80 4.90 6.03 6.05 Access to efficient and independent justice 13.80 7.90 7.90 8.03 4.00 Victims of crime, including compensation to victims of crime 0.00 2.20 1.90 1.11 2.15 Cross-cutting activities

Cross-cutting projects or activities covering all MAF 20.20 22.00 27.50 22.04 35.40 Other activities

Bodies of the Agency and consultation mechanisms 3.30 6.10 5.40 4.54 - Other expenses

Support activities 35.00 39.00 36.40 35.15 38.00

TOTAL 104.4 113.00 115.00 116.00 120

By breaking down the planned human resources into the different thematic areas and support activities we can observe the most ‘human intensive’ areas. First, the greatest amount of human resources is dedicated to the supportive activities of the Agency, as shown for the years 2014, 2015 and 2016. The change from 2013 to 2014 may be explained by the 2014 organisational restructuring of the Agency. Furthermore, in 2013, ‘support activities’ were included in Annex II ‘Administrative activities’ (for a total of 35 staff). Second in human resource is the cross-cutting areas which involve more than one thematic area. In line with the budgetary findings, increased activity from a human resources perspective appear for the areas of Discrimination, Asylum, immigration and integration of migrants and Access to efficient and independent justice. The trend suggesting an increase in the activities relating to Asylum, immigration and integration of migrants is confirmed, in the planning for 2016.

2.2.4. Planning

Activities and projects carried out by the Agency are developed based on a five-year Strategic Plan and Annual Work Programmes prepared two years in advance, which take into consideration key performance indicators and information gained through stakeholder consultations. In addition, and as detailed in the Agency’s Founding Regulation, the Agency is required to lay down its thematic areas of activity in Multi-Annual Frameworks, defining the limits of the work of the Agency. Due to the political significance of the Multi-annual Framework, the European Council, after consulting with the European Parliament, adopts a proposal from the Commission.

The Multi-Annual Framework

 ‘The work of the EU Agency for Fundamental Rights (FRA) is based on its founding regulation

and a Multi-Annual Framework (MAF). Whereas the former defines its overall mandate, including its objective and tasks, the latter lists the thematic areas in which FRA should carry out these tasks, without prejudice to responses of the Agency to requests from the European Parliament, the Council of the European Union or the European Commission outside these thematic areas, provided its financial and human resources so permit. The MAF is revised every

five years, with the current MAF covering the years 2013 to 2017.’ 37

37 Opinion of the Management Board of the EU Agency for Fundamental Rights on a new Multi-Annual Framework (2018–2022) for the agency

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

The Agency similarly holds stakeholder consultation processes to involve stakeholders in the drafting of the Multi-Annual Framework, the Strategic Plan and Annual work programmes. Stakeholders’ views and suggestions are reflected in these documents and are then taken into consideration in the planning of FRA activities.

In 2017, the Programming Document 2017–2019 replaces the Annual Work Programme as the Agency’s single annual planning document. The Programming Document is a planning instrument streamlined for all EU agencies, including the Annual Work Programme and multi-annual planning on strategic and thematic levels. This was introduced due to new programming requirements for EU agencies according to the EU’s Framework Financial Regulation.

Based on this procedure in January 2016 the Agency submitted the first Programming Document 2017– 2019 to the European Commission for its opinion and the Management Board adopted the final version of the Programming Document 2017–2019 at its meeting in December 2016. In addition, in December 2016 the Management Board endorsed FRA’s Draft Programming Document 2018–2020, which was submitted in January 2017 to the European Commission for its opinion.

The Programming Document consists of three sections:

 An overview of the Agency illustrating its mandate, working methodology and core activities.

 The multi-annual Programming 2017–2019, which describes and summarises the Agency’s

objectives. It provides an overview of what is expected in terms of impact according to the Agency’s six strategic priorities and thematic objectives as well as all implemented actions and initiatives.

 The annual Work Programme gives details on the multi-annual and annual projects that the

Agency plans to undertake in 2017.

Regarding the planning of specific projects of the Agency, MATRIX is the main project management tool used by the Agency, which enables research, production, communication and stakeholder engagement activities to be planned, recorded and monitored per relevant milestones.

Similarly, the Agency’s Annual Work programme monitoring report was developed as a new tool for planning, monitoring and reporting of the FRA thematic areas and aims to provide a periodical overview of the Annual Work Programme (AWP) implementation.

When planning projects, the Agency has adopted a Performance Measurement Framework that contains the indicators that are used to measure FRA’s performance. The PMF contains project-level indicators describing the planned output of each project and short-term, long-term and aspirational indicators. The PMF is linked to both the Strategic Plan and AWP.

The Agency’s integrated planning and project management is detailed below.

Year FRA planning and project management: an integrated approach

N-3 November/December

Schedule of  an internal consultation on the FRA’s multi-annual activities from year N-1 to N projects to be  All carry-over projects and their activities and budgets are shared with the

carried over into Management Team and the Director signs off on these carry-over projects for

year N year N.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Year FRA planning and project management: an integrated approach

N-2 From January to March:

From the  Priorities for the year N are identified and a drafted Programmatic Note of new

Programmatic activities, cross-thematic projects and cross-cutting activities for the year N note to the Draft April

Programmatic  The Programmatic note is submitted to the Management Board

document May

 The Management Board will discuss the programmatic note for the year N

From June to July:

 Based on Management Board discussion, the programmatic note is updated and

consulted with FRA stakeholders on which basis the new project fiches for the

AWP year N are drafted

From August to October

 The Draft Programming Document N-N+2, which includes the annual work

programme for year N, is prepared through the following steps:

 Planning prepares the draft of the Programming Document including all sections

and the AWP

 The operational services will update all relevant sections of the document  The Draft Programming Document is shared with the Management Team and the

Director will sign off From October to November

 The formal opinion from the Council of Europe and Scientific Committee is

requested and integrated into the Draft programming document November

 Final Draft Programming Document is signed off by the Director and submitted to

the Management Board, together with opinions by COE and the Scientific

Committee

December

 The Management Board will endorse the Programming Document for submission

to the EC Year N-1 January

Towards the  Final draft Programming Document is sent to the European Commission,

Final Draft European Parliament and European Council no later than 31 January. Programmatic From February to April

document and  In case informal comments are received from the European Commission, the

the definition of Draft Programming Document is fine-tuned. detailed May

projects’ plan  Management Board discussion on the updated Programming Document of year N

July

 The formal opinion of the European Commission on the Programming Document

of the Year N is sent to FRA From September to October

 The Programming Document is revised based on the formal opinion from the

European Commission From October to November

 The revised Programmatic document is submitted to the Management Board for

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Year FRA planning and project management: an integrated approach

final adoption.

 The list of Project Managers is prepared by the Heads of Departments and shared

by Planning with all departments.

 The detailed project plan of all projects/activities as included in the Annual Work

Programme of Year N is defied and inserted into MATRIX by the Project

managers.

December

 Management Board adopts the revised Programming Document N-N+2, which

includes the Annual Work Programme for Year N.

Year N From January to December

Systematic  The Annual Work Programme implementation is monitored and progress is

monitoring and reported.

reporting of

AWP

implementation

Year N to January Year N

N+3  The detailed project plan of a project/activity included in the Annual Work

Step by step Programme for Year N is defined, signed off by the Director and inserted in

procedure to MATRIX.

implement FRA January Year N to N + 3

projects  The project/ activity is implemented according to the adopted plan

Year N+3 From January to December

 Based on FRA implementing rules, ex-post evaluation is undertaken after project.

Future of the FRA

Whilst the Agency’s Founding Regulation defines FRA’s mandate, the Multi -Annual Framework identifies the specific thematic areas for FRA’s activity for a five-year period. The Multi-Annual Framework is adopted by the Justice and Home Affairs Council of the European Union, at the proposal of the European Commission and after consulting the European Parliament.

To this end, in September 2015, FRA carried out a stakeholder consultation in order to gather external input on its new Multi-Annual Framework for the 2018–2022 period. 38 In the consultation, stakeholders were asked to assess the current thematic areas and state whether the same areas should continue to be part of the new framework. The stakeholders were also asked to list any thematic areas that should be added and provide additional general comments on FRA’s activity.

As a result of the stakeholder consultation, the Opinion of the Management Board of the EU Agency for Fundamental Rights on a new Multi-Annual Framework (2018–2022) was published. 39 The opinion summarised the findings of the stakeholders’ consultation and made further recommendations with

38 Consultation of stakeholders for the Multi-Annual Framework 2018–2022, October 2015, Fundamental Rights Agency

39 Opinion of the Management Board of the EU Agency for Fundamental Rights on a new Multi-Annual Framework (2018–2022) for the agency,

February 2016, Fundamental Rights Agency

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

regard to FRA’s activities and new thematic areas, as well as addressing some issues that the Agency has been facing. The document will feed into the Agency’s future work as well as its strategic approach.

The Management Board made the following recommendations, inter alia:

 That the Agency form a new thematic area having two components: ‘Roma integration, and social

inclusion.’ This would extend the Agency’s work to the ‘increasingly relevant debate on social inclusion and vulnerability’.

 That the Agency expand its activity to judicial cooperation in criminal matters – which so far is not

covered by the Multi-Annual Framework. This will allow FRA to support the protection of fundamental rights in police and judicial cooperation in criminal matters. This would be in line with the views of the European Parliament, which has characterised such cooperation as a ‘standard EU policy’. The exclusion of police and judicial cooperation in criminal matters stems from the fact that when FRA was founded, the EU operated under the three pillars system and the FRA’s Founding Regulation did not cover the third pillar which is exactly about this thematic area. However, with the Lisbon Treaty coming into force and the consequent abolishment of the three pillars, the Agency finds itself in a new normative environment. To date, FRA has only been involved with these issues by responding to specific stakeholder requests. Based on stakeholder feedback, the European Parliament postulations in this regard and the results of the first external evaluation, the Management Board suggested FRA should be entitled to conduct activities on its own initiative also in the area of cooperation in criminal matters.

 That the Agency, when dealing with the thematic areas defined in its MAF, should take into account

all relevant rights (such as the Charter of Fundamental rights of the European Union, the European Convention for the Protection of the Human Rights and Fundamental Freedoms, etc.). The Charter should be referred to explicitly and prominently in the MAF, taking into consideration the entry into force of the Lisbon Treaty.

 That the MAF should acknowledge that ‘the founding regulation determines permanent tasks –

including the task of raising the general public’s awareness of their fundamental rights as enshrined in the Charter of Fundamental Rights’.

 That the MAF ‘recall that a variety of horizontal obligations under EU primary law cut across all MAF

areas and affect the way in which the Agency addresses the latter. These include the fight against exclusion and discrimination, the promotion of social justice and protection, the protection of the child, gender equality and respect for diversity.’

The Management Board suggested the following Multi-Annual Framework for the period 2018–22:

 racism, xenophobia and related intolerance;

 equality and non-discrimination;

 rights of the child;

 access to justice and victims of crime;

 judicial and police cooperation;

 migration, borders, asylum and integration of refugees and migrants;

 Roma integration, and social inclusion;

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

 information society and, in particular, respect for private life and protection of personal data.

Table 10 shows the thematic areas decided by the FRA in its Multi-Annual Frameworks across its operation. This illustration shows the difference between the MAFs and how the thematic areas have progressed throughout the Agency’s lifespan. Particularly interesting are the changes in the area of Roma Integration, which became an independent thematic area only in the MAF 2013–2017, and it is suggested to continue its activity in the new MAF 2018–2022. In addition, the thematic area of Visa and Border Control was incorporated into the Immigration and integration of migrants, visa and border control and asylum area of the MAF 2013–2017. Lastly, the thematic area of Participation of the EU citizens in the Union’s democratic functioning was not included in the MAF 2013–2017. Interesting changes between the Management Board’s opinion and the Council recommendation for the 2018– 2022 Multi-Annual Framework include the spelling out of the exclusion of criminal matters from the judicial cooperation thematic area. Furthermore, the Management Board’s opinion was to separate the integration of Roma from social inclusion (which could also cover other groups of persons). This proposal was rejected by the Council in the re-formulation of the thematic area into ‘integration and social inclusion of Roma’.

Table 10: Evolution of Multi-Annual Frameworks across five-year periods and the opinion for the new Multi-Annual Framework for 2018–2022

Multi-Annual Framework Multi-Annual Framework Opinion for Multi-Annual Council recommendations 2007–2012 2013–2017 Framework 2018–2022 2018–2022

 Racism, xenophobia  Racism, xenophobia  Racism, xenophobia  Racism, xenophobia and

and related and related and related related intolerance. intolerance. intolerance. intolerance.

 Discrimination based  Discrimination based  Equality and non Equality and

on sex, race or ethnic on sex, race, colour, discrimination. discrimination based on origin, religion or ethnic or social any ground such as sex, belief, disability, age origin, genetic race, colour, ethnic or or sexual orientation features, language, social origin, genetic and against persons religion or belief, features, language, belonging to political or any other religion or belief, minorities and any opinion, membership political or any other combination of these of a national opinion, membership of grounds (multiple minority, property, a national minority, discrimination). birth, disability, age property, birth, or sexual orientation. disability, age or sexual orientation, or on the

grounds of nationality.

 The rights of the  Rights of the child.  Rights of the child.  Rights of the child.

child, including the protection of children.

 Compensation of  Victims of crime,  Access to justice and  Victims of crime and

victims. including victims of crime. access to justice.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Multi-Annual Framework Multi-Annual Framework Opinion for Multi-Annual Council recommendations 2007–2012 2013–2017 Framework 2018–2022 2018–2022

compensation to .

 Access to efficient  Access to justice;

and independent justice.

• - 

   Judicial cooperation,  Judicial and police  Judicial cooperation,

  except in criminal cooperation. except in criminal matters. matters.

 Asylum, immigration  Immigration and  Migration, borders,  Migration, borders,

and integration of integration of asylum and integration asylum and integration migrants. migrants, visa and of refugees and of refugees and

border control and migrants; migrants;

 Visa and border asylum; control.  Roma integration;  Roma integration, and  Integration and social

social inclusion; inclusion of Roma;

 Information society  Information society  Information society  Information society and,

and, in particular, and, in particular, and, in particular, in particular, respect for respect for private respect for private respect for private life private life and life and protection of life and protection of and protection of protection of personal

personal data. personal data. personal data. data.

 Participation of the - -

EU citizens in the Union’s democratic functioning.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

• 3. 

  Evaluation context

  The FRA’s intervention logic

This section introduces the intervention logic of the Agency, its mandate according to the Founding Regulation as well as the way it is organised, including planning and reporting mechanisms. It is

presented in the form of an intervention logic, setting out: the needs, objectives, inputs, activities and effects. The aspects falling under each of these themes will be explained further in Table 11.

Table 11: Outline of our approach to the intervention logic of the European Union Agency for Fundamental Rights

European Union Agency for Fundamental Rights: intervention logic

The intervention logic is made up of the following building blocks, which are defined here and will be discussed throughout this chapter:

Needs – The overarching need in terms of fundamental rights in the EU is to ensure that the

fundamental rights needs for anyone located in the European Union are met. While the Agency’s

role is not to address these needs, its objectives should aim to have some impact on these needs.

Objectives – The overarching objective of FRA is set out in Article 2 of the Founding Regulation. In operational documents these overarching objectives have been categorised into general, specific and operational objectives which all intend to address the identified needs/problem.

Inputs – this term covers the financial and human resources available to the Agency in order to achieve the stated objectives. This is covered in section 2.3 rather than in this section.

Activities & Processes – the processes, tools and actions that are implemented by the Agency using the inputs available in order to achieve the intended objectives. This is also covered in section 2.3 rather than in this section.

The spectrum of effects, consisting of:

Outputs – the direct products of the Agency’s activities; outputs are easily quantifiable and attributable

to the Agency’s activities.

Results – the direct and immediate effects brought about by the Agency’s activities; results are less

easily quantifiable and attributable to the Agency’s activities than outputs, but more so than impacts. Results are also specifically relevant to the direct stakeholders of the Agency’s activities.

Impacts – the effects of the Agency’s activities beyond the immediate effects (i.e. the impact on the

wider needs); these can be direct or indirect, intended or unintended effects.

Although the Commission’s ‘Roadmap’ on the Common Approach to EU decentralised agencies named the elaboration of common evaluation guidelines for EU agencies as an action to be undertaken, 40 no specific guidance, including within the 2015 Better Regulation Guidelines, is provided as yet on the evaluation of EU Institutions, bodies and agencies.

40 European Commission (2012) Roadmap on the follow-up to the Common Approach on EU decentralised agencies, p. 8

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Commission evaluation guidelines (as part of the Better Regulation Guidelines) are considered to be generally applicable to Agencies inasmuch as the scope of the guidelines is concerned, which focuses largely on ‘retrospective’ evaluations. Better Regulation Guidelines have to be considered as the overarching framework within which the ‘Evaluation handbook for Agencies’, adopted by the EU Agencies network in 2014, should be situated. The handbook is a practical evaluation tool for EU decentralised Agencies, based on the combined experience of theoretical and practical methodologies

for assessing performance, as used at present by the Agencies.

In addition, the external evaluation of FRA in 2012 did not make any recommendation regarding the creation of an evaluation framework for the Agency. Instead it suggested that FRA should review its priorities, refocus its activity around issues relevant to a number of Member States and provide a strategy for managing increasing demands of work, in order to respond to pertinent needs on

fundamental rights issues within available resources.

With the above in mind, the intervention logic aspects are described below using the terminology of the

2015 Better Regulation Guidelines 41 and the terminology specific to the Agency in its programmatic

documents.

3.1.1. Needs

This section provides an overview of the fundamental rights needs/problems/issues in the EU for the period covered by the evaluation (2013 to 2017) and how these needs changed during this period.

In line with Article 2 of the Founding Regulation, this assessment focuses on the needs of the target groups of the FRA (i.e. the duty bearers), in terms of the type of assistance they need, and on which

thematic areas. These duty bearers are:

 EU Institutions, bodies, offices and agencies; and

 EU Member States (as well as candidate countries and potential candidate countries).

However, taking into account the overall objective of the Agency which is, according to Article 2 of the Founding Regulation, to provide assistance and expertise for the purpose that duty bearers ‘fully respect fundamental rights’ of individuals, information on the needs of the rights holders has been added to this section. Although rights holders are not the direct target group of the FRA, they are the target group of

the duty bearers to whom the Agency must provide assistance.

41 European Commission (2015) Better Regulation Guidelines & Better Regulation ‘Toolbox’

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Figure 3: Impact of the FRA on rights holders

FRA Duty Bearers Right holders

Indirect impacts

Information on their needs will provide a deeper understanding of the needs the duty bearers are aiming to address and at the same time provide a better understanding of the context in which the FRA operates. Therefore, the needs of the rights holders are also relevant inasmuch they inform the needs of the duty bearers. It is important to note, however, that the FRA will not be evaluated against the impact it has on rights holders, but solely on the ways in which it addresses the needs of the duty bearers.

The data sources for understanding the needs of the FRA target groups was derived by undertaking desk research, as well as through an internal and external survey and interviews with:

 The duty bearers themselves:

o EU level: officials of EU Institutions, bodies, offices and agencies; and

o Member States level: National Liaison Officers (NLOs); National Parliamentary Focal Points;

National Human Rights Institutions (NHRI).

 Bodies and staff of the EU Fundamental Rights Agency (FRA);

 The representatives of organisations promoting fundamental rights in general or for a specific group

(people with disabilities, Roma, LGBTI, etc.) such as international organisations responsible for

protecting human rights (e.g. the Council of Europe – CoE and the United Nations High

Commissioner for Refugees – UNHCR) and civil society and NGOs.

3.1.2. Objectives

The general objective of the FRA is set out in Article 2 of the Founding Regulation. The Multi-Annual Framework 2013–2017 (MAF) identifies the thematic areas in which the FRA carries out its tasks (set out in Art. 4 of the Regulation) on its own initiative. This is done on the basis of the strategic priorities and thematic objectives defined by the Management Board in the Strategic Plan. The Strategic Plan 2013–2017 contains different levels of objectives, at the thematic and strategic level, including expected impacts, expected results, planned activities and actions. In addition, the Mid-Term Review of the Strategic Plan 2013–2017 revised specific objectives and expected results, as well as the main operational focus until the end of 2017 for each strategic priority and thematic objective. Finally, every year Annual Work Programmes (AWP) were adopted.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

As part of the effort to align the Strategic Plans and the Annual Work Programmes, from 2016 all EU agencies had to draft a Single Programming Document in which both components had to be included. In 2016, the first Single Programming Document was drafted for FRA for the period 2017–2019, to ensure a more consistent approach between annual and multi-annual programming and define the common elements on the work programme (providing a common terminology for objectives, activities, outputs, results, impacts, etc.). The Single Programming Document building on the period 2013–2015 set out the plans as well as objectives, expected results and priorities for each specific priority and for each thematic objective. Moreover, the document explains the link between strategic and thematic objectives with the Performance Measurement Framework (PMF). The second Single Programming Document will be drafted in 2017.

As shown in Figure 4, these different levels of objectives have been translated into:

 General objective – Article 2 of the Founding Regulation stipulates that the objective of FRA is to

assist the EU and its Member States to respect fundamental rights. In order to achieve this objective, Article 4 of the Founding Regulation establishes eight specific tasks (including the collection, analysis and dissemination of information, undertaking surveys, formulating and publishing conclusions and opinions on specific thematic topics, for the Union institutions and the Member States, etc.). Pursuant to Article 5 Para 3 of the Founding Regulation, FRA, when acting on its own initiative, is only allowed to carry out the tasks listed above within the thematic areas laid

down in the Multi-Annual Framework (MAF). 42 The nine thematic areas for the period 2013 to

2017 are mentioned in Figure 4. It should be noted that FRA is allowed to go beyond these thematic areas when carrying out its tasks (scientific research and surveys, preparatory studies and feasibility studies, as well as when it is developing conclusions and opinions on specific thematic topics) at the

43

request of the European Parliament, the Council or the Commission.

 Thematic objectives – The thematic objectives are set out in the Strategic Plan for 2013–2017. Each

thematic objective in the Strategic Plan is accompanied by a set of specific objectives, key activities and expected results. Moreover, the Annual Work Programmes (AWP) sets out annual objectives for each of the nine thematic objectives listed in the Strategic Plan. Finally, the Annual Activity Reports do not report achievements by the nine thematic objectives, but for each of the thematic areas listed in the MAF. In 2016, progress made towards the achievements of the thematic priorities in the Strategic Plan was assessed in the Mid-Term Review of the FRA Strategic Plan 2013–2017. Although the Mid-Term Review ‘largely reaffirmed the existing strategy’, it also set revised objectives and expected results.

 Strategic priorities and objectives – The strategic priorities are also set out in the Strategic Plan for

2013–2017. They cover a range of activities and goals that the FRA is aiming to achieve over the defined period. As shown in Figure 4, the strategic priorities are more process oriented and relate to how the Agency is to better achieve its general objective and thematic objectives (i.e. the means to an end / the mechanism to achieve the other objectives). The Annual Work Programmes for 2012, 2013 and 2014 do not mention the strategic priorities and therefore do not include any further annual objectives under each Strategic Priority (as is the case for the thematic objectives). However, since 2015 the Annual Work Programme includes a specific section on the FRA Strategic priorities for 2013–2017. The Annual Activity Reports do report on the achievements under each

42 Council Decision No 252/2013/EU establishing a Multi-annual Framework for 2013–2017 for the European Union Agency for Fundamental

Rights.

43 Article 5 (3) Regulation in conjunction with Article 4(1)(c) and 4(1)(d) Regulation.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

strategic priority. As with the thematic objectives, in 2016 progress made towards the achievements of the strategic priorities in the Strategic Plan was assessed in the Mid-Term Review

of the FRA Strategic Plan 2013–2017 and the strategic priorities were largely confirmed.

 Operational objectives: No specific operational objectives were set in the Regulation, Multi-Annual

Framework or Strategic Plan. The Strategic Plan 2013–2017 does provide some level of operational objectives (e.g. ‘key actions’ for each strategic objective and ‘key actions’ for each thematic

objective). Moreover, the Annual Work Programmes list the proposed projects for a particular year under each thematic area. Therefore, for the purpose of this study the operational objectives relate to the projects of the FRA. Given that the Agency has adopted activity-based monitoring and

budgeting, the objectives set out in the Performance Measurement Framework will be used to assess the projects on their effectiveness.

Each of the different types of objectives are presented in Figure 4.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Figure 4: Different levels of FRA’s trends

es g

tiv ”The objective of the Agency shall be to provide the relevant institutions, bodies, offices and d in ) jec agencies of the Community and its Member States when implementing Community law with u n ion

Fo lat assistance and expertise relating to fundamental rights in order to support them when they

al Ob gu take measures or formulate courses of action within their respective spheres of competence

er rt . 2, Re to fully respect fundamental rights ”

Gen (A

ives ) Access to Informati

je ct lan justice Victims of on (incl. crime ic Ob e gi

c P

judicial (incl.

Society Roma Rights of Discrimi Immigra FRA

rat cooperati compens

(incl. Integration the child nation tion and Racism Asylum surveys

emat (St on) ation)

privacy and DP)

Th

tives ) lan Enhancing Enhancing Identifying Developing Improving the

b jec FRA’s FRA’s trends over timely and

Planning FRA impact of FRA

c O e gi

c P

contribution at contribution at time targeted

work and

communication evaluating

e gi rat EU level nat. level

responses to and awareness

(St emergencies

impact raising

St rat

al

at ion tives The operational Objectives are set out in the Strategic Plan and are specific to both the thematic and strategic objectives

er jec

O p O

b 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

3.1.3. Effects (Outputs; Results; Impacts)

In this intervention logic, the effects of the Agency mirror the different levels of objectives developed in the previous section and can be divided into three different aspects, namely outputs, results and impacts. Impact itself can be considered to be the final level of effect and reflect the ways in which the needs identified have been addressed, both for the target groups to which the Agency provides its assistance (i.e. the EU Institutions, bodies, offices and agencies and the EU Member States) and beyond (and potentially indirectly) to rights holders in the European Union. Impact itself can be broken down

into short-term, long-term and aspirational impacts.

Here the different levels of ‘effects’ are explained by illustrating how they are measured in general and in regard to specific FRA projects. Each FRA project is, at the planning stage, given a set of outcome objectives as well as short-term, long-term and aspirational impact indicators that it needs to fulfil. After the completion of the project, the measurement indicators assist the Agency in assessing the effects of the project on its output requirements and also on how it promotes the entire organisation short-term,

long-term and aspirational objectives.

Outputs

Outputs are the first level of effects; they are defined as the direct products of the FRA’s activities. Outputs are the most easily identifiable and quantifiable effects and are also the easiest to attribute to

the intervention – they relate to the Operational Objectives of the Agency.

The Annual Activity Reports include a selection of output indicators usually aggregated at a thematic area level. Although the number and nature of the indicators that FRA uses to monitor its activity vary annually depending on the tasks undertaken, of particular interest are the FRA core indicators that remain the same throughout the evaluation period. Examples of output indicators as extracted by the Agency’s Annual Activity Reports for the years 2013, 2014 and 2015 44 include for each thematic area among others: publishing of thematic reports and handbooks; providing assistance and expertise to EU Institutions and Member States; and developing effective networks with key partners and agency

network.

The outputs of specific projects can be found in the MATRIX information system. In order to illustrate the outputs indicator measurement for FRA’s projects and in an effort to reconstruct the intervention logic of the Agency we present some of the outputs of the FRA project: Fundamental rights safeguards and remedies in light of surveillance by national authorities below. The output measurements for the

project include, inter alia:

 Publication of report on surveillance by national intelligence services examining fundamental rights

safeguards and remedies in the 28 Member States of the EU;

 The preliminary findings for the report were presented to the European Parliament and

consequently the Parliament published a resolution addressing the need to safeguard citizens’ fundamental rights;

 FRA participated in an inter-parliamentary committee meeting on democratic oversight of national

intelligence services at the European Parliament.

44 The Annual activity report for 2016 has not been published yet. The Annual activity reports are usually published at the start of the following

year – in this case in 2017.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

These and other achievements are used to assess the project’s success and contribute to the aggregated effects of the thematic area that the project belongs to.

Results

Results are the second level of effects; they are defined as the direct and immediate effects brought about by the FRA’s activities. Often, the results relate to the effect of an activity on the target group or stakeholders involved.

In line with FRA’s Performance Measurement Framework, ‘results’ is a broad category of impacts that include the short-term and long-term impacts. The same aggregation processes are followed in measuring these effects: projects may be assigned a number of short-term and long-term objectives that contribute to the overall thematic area effect. Furthermore, FRA’s thematic areas and their priorities aggregate to a higher level, developing the strategic priorities that FRA has established in its Strategic Plan for 2013–2017.

A summary of the results achieved in a given year can be found in FRA’s Annual Activity Reports. A specific example of these effects is presented below, in the context of FRA’s strategic priority: Enhancing FRA’s contribution to processes at EU level. By way of illustration, in 2016, the Agency cooperated with the EU Institutions in the following ways, among others:

 The FRA contributed to, and participated in, 19 meetings – including hearings – at the European

Parliament.

 The FRA Cooperated with the European Parliament’s PETI Committee, the European Commission

and the European Ombudsman in the EU Framework to promote and monitor the UN Convention on the Rights of People with Disabilities (CRPD).

 The FRA’s data on Roma contributed to the Commission’s and the Council’s assessment in the

context of the European Semester and was prominently referenced in the Court of Auditor’s report on Roma inclusion of June 2016. 45

 Documents paving the way for the Union’s accession to the Istanbul Convention (Council of Europe

Convention on preventing and combating violence against women and domestic violence) referred extensively to work carried out by the Agency and in particular the Violence Against Women survey (see case study 1).

These achievements show the effect of the Agency’s work beyond the operational outputs of its projects and activities as they directly involve its stakeholders and network. (A more detailed explanation of the difference between results and outputs is given in section 4.1.2.)

Impacts

Impacts are the third and highest level of effects; they are defined as the effects of the Agency’s activities beyond the immediate effects. However, impacts are the most difficult effects to quantify and identify; and it is also difficult to attribute any effect to a specific activity due to the influence of external factors. Furthermore, impacts are closely related to the general objectives outlined above and beyond this, to the needs the Agency is designed to address.

45 European Court of Auditors, Special report no 14/2016: EU policy initiatives and financial support for Roma integration: significant progress

made over the last decade, but additional efforts needed on the ground, June 2016.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

FRA has defined a number of short-term, long-term and aspirational indicators in Annual Work

Programme 2016 to evaluate the impacts of the FRA project on ‘Biometric data in large IT borders,

immigration and asylum databases – fundamental rights concerns.’ This included, among others:

 Number of references to FRA’s findings and opinions in EU and Member State policies, reports, and

legislative preparatory work;

 Number of references by civil society actors to FRA findings;

 Stakeholders who agree that FRA conclusions and recommendations inspire or influence EU policy

and legislative changes

These indicators and related indicators can be generalised to measuring FRA’s impact on a higher level

(i.e. measuring developments in the area of fundamental rights and identifying the links to the Agency’s activities).

Policy context

This section sets out the different actors FRA cooperates with within the fundamental rights galaxy. The different actors have different mandates, and therefore require a varying level of cooperation with FRA. FRA operates from a unique position, responding to the needs of both Member States as well as international organisations/cross-government needs. According to their function, the cooperating actors are clustered into six categories: policy-making, research, watchdog, enforcement, policy implementation and funding. This section provides an overview of the organisations and types of organisations with which the FRA cooperates and the context in which it operates, as well as more detailed examples of the type of cooperation and examples of such cooperation.

Figure 5: The FRA and its external stakeholders

Cross-government / international level Member State level

EU institutions United and bodies Nations • EP • ECoR • UN OHCHR • ILO • Council of • EESC • UNDP • UNICEF the EU • European • UNECE • UNHCR • EC Ombudsma • UNODC • IOM National Human Rights Government • EU agencies n Bodies

PM PM • Equality Bodies

• NHRIs

PI W PM E F W • Ombudsperson

institutions OSCE Council of Europe

Comprehensive approach to Advocates freedom of

security that encompasses expression and of the media, FRA W E R PI PM

politico-military, economic freedom of assembly, and environmental, and equality, and the protection National Parliaments Administration human aspects. of minorities. National Parliamentary Focal Points meet annually to

R W R PM E evaluate how cooperation is

Equinet European EEA & Norway working in practice and to Network of Grants listen to needs of national

European National Human parliaments.

Network of Rights Institutions Contribute to

Equality Bodies reduce economic PM E PI

Brings together 40 and social NHRIs across disparities and to wider Europe. strengthening bilateral relations.

R R F

Enforcement Policy making Watchdog Research Funding Policy implementation

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Examples of cooperation between the FRA and other organisations operating in the field of Fundamental Rights are presented below.

3.2.1. National level

Member States of the EU

National Parliaments

FRA regularly cooperates with the national parliaments of EU Member States.

It does this by presenting the findings from FRA projects and by providing evidence-based advice to national parliaments. FRA has established a network of parliamentary focal points taking inspiration from the National Liaison Officers (NLOs) in order to gain a broader understanding and increase its

impact at national level.

The assistance and expertise provided by the Agency helps the Member States to fully respect fundamental rights when implementing EU law. This complements the Agency’s cooperation with the

EU Institutions.

For example, as part of its Roma project, the Agency is working closely with EU Member States to develop methods that measure progress on Roma integration across the EU. In another area, FRA collects evidence of the extent to which lesbian, gay, bisexual and transgender people’s rights are respected in the EU to help governments develop policies in line with the EU Charter of Fundamental

Rights, which prohibits discrimination based on sexual orientation.

National Liaison Officers

To establish a close working relationship with FRA, each Member State appoints a government official who works directly with the FRA as a National Liaison Officer.

National Liaison Officers (NLOs) are the main contact points for the Agency in Member States. Continuous exchange and regular meetings help to create a strong working partnership, which

guarantees that information is shared and national needs are recognised and addressed.

NLOs have the opportunity to submit opinions on FRA’s draft annual work programmes. This gives the Agency a greater understanding of the type of fundamental rights analysis and assistance the Member States need to shape informed policy. The agency regularly informs NLOs about research findings, upcoming reports and activities, while NLOs provide feedback on the expediency and relevance of the Agency’s work. NLOs also help to communicate FRA’s work to the relevant national ministries,

departments and government bodies.

FRA maintains regular contact with government experts in national, regional and local administrations and organises meetings with them in the context of specific projects. 46

National Human Rights Institutions

FRA works closely with national human rights institutions (NHR/is), Equality Bodies (EB) and National Ombuds institutions (NOI), through annual meetings and by involving selected bodies in the design and implementation of FRA projects, as well as through cooperation activities related to the dissemination

and follow-up of FRA findings. 47

46 http://fra.europa.eu/en/cooperation/eu-member-states

47 http://fra.europa.eu/en/cooperation/national-human-rights-bodies

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

3.2.2. Cross-government/international organisations

EU Institutions and other bodies

FRA may issue opinions and conclusions to EU Institutions and Member States on specific thematic topics. Moreover, the European Parliament, Council or Commission can request the Agency to deliver

opinions on EU legislative proposals ‘as far as their compatibility with fundamental rights are concerned’. This specific task contributes to the Agency’s overall objective to support EU Institutions

and Member States to fully respect fundamental rights. 48

European Parliament

The European Parliament is one of the key institutions of the European Union. It acts as a co-legislator, sharing the power to adopt and amend legislative proposals and decide on the EU budget. The

Parliament also supervises the work of the European Commission and other EU bodies. The Committee on Civil Liberties, Justice and Home Affairs (LIBE) is of particular importance, as it is responsible for the

vast majority of the legislation and democratic oversight of Justice and Home Affairs policies. Whilst doing so, it ensures the full respect of the Charter of Fundamental Rights within the EU, the European

Convention on Human Rights and the strengthening of European citizenship. 49

European Commission

The FRA works closely with the European Commission, with DG Justice and Consumers and DG Migration and Home Affairs. Two Commission representatives are part of the FRA’s Management Board.

FRA staff are in constant contact with the relevant departments at the Commission. In this way, the Commission can readily draw on the Agency’s assistance and expertise when developing, implementing

and evaluating EU policies and legislation. Further cooperation with the Commission is reported throughout this report.

European Council

The Council plays an important role for the FRA, being in particular in charge of adopting the Agency’s Multi-Annual Framework. The results of FRA’s data collection and research feed into the discussions of

relevant Council preparatory bodies, particularly the Working Party on Fundamental Rights, Citizens’ Rights and Free Movement of Persons (FREMP), to which the Agency presents its Annual report on

fundamental rights every year.

Council of Europe

‘The Council of Europe advocates freedom of expression and of the media, freedom of assembly, equality, and the protection of minorities. The Council of Europe helps Member States fight corruption

and terrorism and undertake necessary judicial reforms. Its group of constitutional experts, known as the Venice Commission, offers legal advice to countries throughout the world. The Council of Europe

48 http://fra.europa.eu/en/cooperation/eu-partners

49 http://ennhri.org/Cooperation-with-EU-Institutions

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

promotes human rights through international conventions. It monitors Member States’ progress in

these areas and makes recommendations through independent expert monitoring bodies.’ 50

FRA and the Council of Europe work closely together, whilst ensuring complementarity and added value. The Council of Europe participates in the Management Board of FRA. The relationship between CoE and FRA predominantly involves the exchange of information. FRA collects data on fundamental rights topics that fulfil an information need of the CoE and the data is used as the starting point of the policy-making

work of CoE.

The FRA cooperates closely with the following bodies of the Council of Europe:

 Parliamentary Assembly of the Council of Europe  European Court of Human Rights  Commissioner for Human Rights  Directorate General Human Rights and Rule of Law  Directorate General Democracy

Civil society and the Fundamental Rights Platform

‘The Fundamental Rights Platform (FRP) is FRA’s channel for cooperation and information exchange with civil society organisations active in the field of fundamental rights at the national, grassroots, European

or international level.’ 51

As part of the platform, FRA cooperates with:

 non-governmental organisations dealing with human rights

 trade unions

 employers’ organisations

 relevant social and professional organisations

 churches, religious, philosophical and non-confessional organisations

 universities

 other qualified experts of European and international bodies and organisations.

United Nations

United Nations Office of the High Commissioner for Human Rights (UN OHCHR)

‘As the principal United Nations office mandated to promote and protect human rights for all, OHCHR leads global human rights efforts and speaks out objectively in the face of human rights violations worldwide. OHCHR provide a forum for identifying, highlighting and developing responses to today’s human rights challenges, and act as the principal focal point of human rights research, education, public

information, and advocacy activities in the United Nations system.

50 http://www.coe.int/en/web/about-us/values

51 http://fra.europa.eu/en/cooperation/civil-society/about-frp

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

‘Since Governments have the primary responsibility to protect human rights, the High Commissioner for Human Rights (OHCHR) provides assistance to Governments, such as expertise and technical trainings in the areas of administration of justice, legislative reform, and electoral process, to help implement international human rights standards on the ground. OHCHR also assist other entities with responsibility

to protect human rights to fulfil their obligations and individuals to realize their rights.’ 52

The cooperation between FRA and the OHCHR involves regular exchange of information on projects and events, including in the drafting phase of the Agency’s Annual Work Programmes. FRA also participates regularly in the OHCHR’s consultations and workshops on enhancing the cooperation between the UN and regional mechanisms. A more EU-focused interaction takes place with the OHCHR European office

in Brussels.

United Nations Development Programme (UNDP)

‘UNDP works in nearly 170 countries and territories, helping to achieve the eradication of poverty, and the reduction of inequalities and exclusion. We help countries to develop policies, leadership skills, partnering abilities, institutional capabilities and build resilience in order to sustain development

results.’ 53

FRA and UNDP work together on the thematic area of Roma.

In 2011 FRA and UNDP signed a protocol for cooperation in the areas of data collection and research, networking, communication and awareness and capacity development. They coordinate their work and contribute to providing data, analysis and evidence-based advice on the situation of Roma. The surveys, which share core components, yield robust and comparative data on the socio-economic status of Roma alongside data on the enjoyment of their rights in practice. The first results of their combined surveys

are available in the report ‘The situation of Roma in 11 EU Member States – Survey results at a glance’.

United Nations Office on Drugs and Crime (UNODC)

‘UNODC is mandated to assist Member States in their struggle against illicit drugs, crime and terrorism. In the Millennium Declaration, Member States also resolved to intensify efforts to fight transnational crime in all its dimensions, to redouble the efforts to implement the commitment to counter the world

drug problem and to take concerted action against international terrorism.’ 54

FRA contributed to the development of the UNODC-UNECE Manual on Victimization Surveys. The Agency has also contributed to UNODC’s work on hate crimes in the context of the 21st session of the United Nations Commission on Crime Prevention and Criminal Justice. Furthermore, there has been close cooperation with the UNODC particularly within the project ‘Fundamental rights of migrants in an irregular situation in the European Union’ as well as on the project on Treatment of third-country

nationals at the EU’s external borders.

52 http://www.ohchr.org/EN/AboutUs/Pages/WhatWeDo.aspx

53 http://www.undp.org/content/undp/en/home/operations/about_us.html

54 http://www.unodc.org/unodc/en/about-unodc/index.html?ref=menutop

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

International Labour Organisation (ILO)

‘The main aims of the ILO are to promote rights at work, encourage decent employment opportunities, enhance social protection and strengthen dialogue on work-related issues. 55

‘FRA has cooperated closely with the ILO particularly within the project Fundamental rights of migrants in an irregular situation in the European Union as well as on the project on Treatment of third-country

nationals at the EU’s external borders.’

Representatives of ILO also participated in the FRA’s Fundamental Rights Conference on Dignity and rights of irregular migrants.

 United Nations Children’s Fund (UNICEF)

‘UNICEF works in 190 countries and territories to protect the rights of every child. UNICEF has spent 70 years working to improve the lives of children and their families. Defending children’s rights throughout their lives requires a global presence, aiming to produce results and understand their effects. UNICEF was created with a distinct purpose in mind: to work with others to overcome the obstacles that

poverty, violence, disease and discrimination place in a child’s path.’ 56

FRA has been part of the Child Protection Monitoring and Evaluation Reference Group (TWG CP MERG) co-chaired by UNICEF New York and Save the Children. Cooperation is also taking place with the UNICEF Innocenti Research Center in Florence in relation to the children and justice project of the FRA. The Agency has cooperated with UNICEF particularly on the project ‘Fundamental rights of migrants in an irregular situation in the European Union’ as well as on the project on Treatment of third-country

nationals at the EU’s external borders.

United Nations High Commissioner for Refugees (UNHCR)

‘UNHCR, the UN Refugee Agency, is a global organisation dedicated to saving lives, protecting rights and building a better future for refugees, forcibly displaced communities and stateless people. UNHCR strive to ensure that everyone has the right to seek asylum and find safe refuge in another State, with the option to eventually return home, integrate or resettle. During times of displacement, UNCHR provide critical emergency assistance in the form of clean water, sanitation and healthcare, as well as shelter, blankets, household goods and sometimes food. They also arrange transport and assistance packages

for people who return home, and income-generating projects for those who resettle.’ 57

The Agency cooperated with UNHCR within its projects ‘The duty to inform applicants about the asylum procedure: the asylum-seeker perspective’, ‘Access to effective remedies: the asylum-seeker perspective’, ‘Separated, asylum-seeking children in European Union Member States’ as well as ‘Detention of third country nationals in return procedures’. UNHCR also participated in the Fundamental Rights Conference on Dignity and Rights of Irregular Migrants. The FRA is also regularly invited to

present its findings at UNHCR events.

55 http://www.ilo.org/global/about-the-ilo/lang--en/index.htm

56 https://www.unicef.org/about/who/index_introduction.html

57 http://www.unhcr.org/uk/what-we-do.html

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

International Organization for Migration (IOM)

‘The International Organization for Migration (IOM) is an inter-governmental organisation committed to the principle that humane and orderly migration benefits both migrants and society. As the leading global organisation for migration, IOM works with migrants, governments and its partners in the

international community to provide humane responses to the growing migration challenges of today.

‘By promoting international cooperation and dialogue on migration issues, IOM assists in the search for practical solutions to key issues facing migrants and societies alike. The IOM Constitution recognises the link between migration and economic, social and cultural development, as well as to the right of

freedom of movement.’ 58

The Agency has cooperated with IOM within the project ‘Fundamental rights of migrants in an irregular situation in the European Union’ as well as on the project on Treatment of third-country nationals at the

EU’s external borders.

Organization for Security and Co-operation in Europe (OSCE)

‘The OSCE has a comprehensive approach to security that encompasses politico-military, economic and environmental, and human aspects. It therefore addresses a wide range of security-related concerns, including arms control, confidence- and security-building measures, human rights, national minorities, democratization, policing strategies, counter-terrorism and economic and environmental activities. All 57 participating States enjoy equal status, and decisions are taken by consensus on a politically, but not

legally binding basis.’ 59

The OSCE is replicating the FRA’s Violence Against Women (VAW) survey in 10 non-EU Member States, which is a significant development. This is the first time the OSCE has ever undertaken to do a fieldworkbased survey of this nature and on this scale. FRA was asked to be part of the steering committee to

oversee the survey’s development.

Office for Democratic Institutions and Human Rights (ODIHR)

‘The OSCE Office for Democratic Institutions and Human Rights (ODIHR) provides support, assistance and expertise to participating States and civil society to promote democracy, rule of law, human rights and tolerance and non-discrimination. ODIHR observes elections, reviews legislation and advises governments on how to develop and sustain democratic institutions. The Office conducts training programmes for government and law-enforcement officials and non-governmental organizations on

how to uphold, promote and monitor human rights.’ 60

FRA regularly participates in the Human Dimension Implementation Meetings in Warsaw. The topics covered in recent events were:

 Roma – FRA was contributing to the event ‘To make a tangible difference to Roma people’s lives’

 Romani women – FRA cooperates with ODIHR on exploring the theme ‘Women as Agents of Change

in Migrant, Minority and Roma and Sinti Communities in the OSCE Area’.

58 http://unitedkingdom.iom.int/about-us

59 http://www.osce.org/whatistheosce

60 http://www.osce.org/odihr

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

 Hate crime – FRA was part of the expert group for ODIHR's practical guide on ‘Monitoring and

Collecting Hate Crime Data for effective policy and practical responses’.

EEA and Norway Grants

‘The EEA Grants and Norway Grants represent the contribution of Iceland, Liechtenstein and Norway to reducing economic and social disparities and to strengthening bilateral relations with 15 EU countries in Central and Southern Europe and the Baltics.’ 61

FRA and the EEA and Norway Grants agreed on a cooperation framework to better promote and protect fundamental rights across the EU. The cooperation arrangement was adopted by the FRA Management Board and signed by the FRA Director on 15 December 2016. The cooperation between the FRA and the EEA and Norway grant is a new one, not originally foreseen in the Founding Regulation.

The Agency cooperates with the EEA and Norway Grants to complement each other’s work and enable

public authorities and civil society to tackle fundamental rights issues on the ground. They signed a cooperation framework on 15 December 2016. The cooperation focuses on the promotion of Roma

inclusion, supporting responses to hate crime by public authorities and civil society, access to justice and follow-up to FRA’s survey on gender-based violence. The EEA and Norway Grants have a strong focus on strengthening fundamental and human rights, and combating social exclusion and all forms of discrimination in many of their funded programmes. The Grants were also a key partner at numerous FRA flagship events such as the Fundamental Rights Forum and the Fundamental Rights Conference

‘Combating Hate Crime in the EU’.

Important issues

3.3.1. The Common Approach

As the result of the Commission Communication on European Agencies for 2008 62 an Inter-Institutional Working Group (IIWG) was set up to assess ways in which agencies were being managed, and in particular to explore whether there was any scope for the development of a common group to help them improve their efficiency. The aim of the group was, inter alia, to decide the future functioning of the agencies. An evaluation of the decentralised agencies (then 26) was undertaken, examining the agencies’ relevance, rationale, good governance, coherence, effectiveness, efficiency and oversig ht activities. 63 The main findings of the evaluation, in regard to FRA’s activity were, inter alia:

 The rationale of FRA is considered relevant and pertinent to the needs identified

 FRA is governed by a highly effective executive board, thus differentiating the organisation from

many of the other decentralised agencies that do not have an executive board

 FRA can play a crucial role in the creation of legislative policies despite the fact that the Agency

does not belong to the inter-institutional decision-making process.

61 http://eeagrants.org/What-we-do

62 Communication for the Commission to the Parliament and the Council of 11 March 2008: European Agencies – The way forward Sec(2008)323. 63 Evaluation of the EU decentralised agencies in 2009, Ramboll, Eureval, Matrix insight (later called Optimity Advisors)

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

 FRA is value-driven and has a bearing in fundamental rights policies across thematic areas

 FRA successfully performs its distinct tasks

 FRA’s activities and impacts depend on its operations (capacity, quality, stability of human

resources).

The findings of the evaluation for all the agencies suggested that there was a lack of a single legal framework governing the agencies in particular around establishment,

Roadmap for Agencies: 59 operation and closure. In 2012, the Inter-Institutional Working Group’s actions to be addressed by work resulted in a Joint Statement and a Common Approach 64 that Agencies – status of marks the first political agreement on EU agencies and acts as a implementation by FRA’ ‘political blue-print’ guiding future initiatives and reforms of the

decentralised agencies. The Common Approach aimed at introducing ‘coherence, effectiveness, efficiency, accountability and transparency of the decentralised agencies and finding common ground on how to improve their work’. The European Commission’s main objectives for the implementation of the Common Approach were: more balanced governance; improved efficiency and accountability; and greater coherence. Furthermore, the Joint Statement urged the Commission to collaborate with the agencies and present a roadmap on the follow-up to the Common Approach with ‘concrete timetables for the planned

initiatives by the end of 2012’.

As a result, in December 2012 the Commission issued the Roadmap on the follow-up to the Common Approach on the EU Decentralised

agencies. 65 In parallel to the actions listed in the Roadmap, the implementation of the Common Approach required the adaptation of the founding acts of existing agencies following a case-by-case analysis, as well as the adaptation or adoption of other legislative acts (Staff Regulations 66 , Framework Financial Regulation 67 ). However, to date, the FRA’s founding Regulation has not been amended to ensure its alignment with the Common Approach.

The Roadmap comprised 89 actions to be delivered by EU Agencies and Institutions. These initiatives were a direct translation of the Common Approach requirements with set deadlines for implementation.

Of these 89 actions, 59 are to be addressed by the agencies themselves. 68 A detailed overview of each of

the actions and the way in which they have been addressed by the FRA can be found in section 4.4.4. To date, the FRA has completed 50 actions listed in the roadmap, with a further six in progress. An additional three actions are not applicable to the FRA. While information is not available for all decentralised agencies, the FRA is clearly one of the most advanced agencies in the adoption of the

roadmap.

The only actions which have not been implemented relate to governance issues and cannot be addressed without a change to the Agency’s Founding Regulation. These are presented in Table 12.

64 Joint Statement of the European Parliament, the Council of the EU and the European Commission on decentralised agencies, 19 July 2012.

65 https://europa.eu/european-union/sites/europaeu/files/docs/body/2012-12-

18_roadmap_on_the_follow_up_to_the_common_approach_on_eu_decentralised_agencies_en.pdf

66 OJ L 287, 29.10.2013, p. 15

67 OJ L 328, 7.12.2013, p. 42

68 Other actions from the roadmap relate to areas under the competence of the Commission, such as setting up agencies, etc.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Table 12: Common Approach roadmap actions, status and issues in relation to FRA

Action Description Current status Notes / issues

#29 Draw up multi-annual strategic The Multi-Annual Framework The adoption of the MAF programmes or guidelines (MAF) is agreed upon by the by the Council can lead to linked with multi-annual Council on the basis of a issues relating to the resource planning (budget and proposal from the Commission Agency’s independence. staff in particular) and linked and after consulting the Timing issues are also with successive annual work European Parliament prominent. The 2013– programmes 2017 MAF was finally adopted in March 2013.

#89 Develop guidelines on tailored The Management Board is in While the Commission is performance indicators to charge of appointing and has involved in the selection assess the results achieved by the power to dismiss the of the Director, the Directors Director. position’s selection

According to Art. 15 of the procedure and potential Founding Regulation, the three years extension is

Commission shall undertake an ultimately in the remit of evaluation, in particular of the the Management Board.

performance of the director and the Agency’s duties and requirements in the coming years.

#90 Appoint members of the boards Members of the Management The key elements relating in light of their knowledge of board are appointed by to the appointment of the Agency’s core business and Member States and are limited Management board taking into account relevant to serve for a maximum of five members are the managerial, administrative and years independence of the

budgetary skills and limit their According to Art 12 of the Members as well as their

turnover Founding Regulation: ‘The relatively short-term

Management Board shall be tenure (five years)

composed of persons with appropriate experience in the management of public or private sector organisations and, in addition, knowledge in the field of fundamental rights’

Overall, it is therefore clear that the Agency has adopted important elements of the Common Approach, indeed been at the forefront of the introduction of some systems such as Activity-Based Budgeting. The Agency also played a key role by chairing the Performance development network (PDN) in 2014, which led to the elaboration and adoption of the programming documents and annual activity reports now used by the FRA (and other decentralised agencies). Furthermore, the Agency has undertaken both ex-ante and ex-post evaluation of its own activities, once again demonstrating the

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

way in which the Agency has implemented the Common Approach. However, this is not to say that all elements of the Roadmap have been implemented, in particular because of the specificity of the

Agency, whose core independence does not allow for all actions to be undertaken.

Furthermore, while the actions of the roadmap are clearly important, so is the spirit of the Common Approach, in particular in the areas of more balanced governance; improved efficiency and accountability; and greater coherence. As such, areas discussed further in the answers to the evaluation question below would need to be amended in order to ensure that the Agency is fully in line with the

Common Approach, in particular:

 a simplification of the approval of the Multi-Annual Framework;

 the clarification of the mandate of the members of the Management Board;

 streamlining the selection procedure for the Director, and potentially amending issues relating to

his or her term in office.

In order for the Agency to be fully aligned with the Common Approach and to be as efficient as possible while ensuring its independence, a change in the Founding Regulation would be necessary.

3.3.2. Recommendations from the first evaluation of the FRA (2012)

The recommendations for actions by the Agency from the first evaluation are illustrated in the table below.

Recommendations by the first external evaluation of the European Union’s Fundamental Rights Agency 2012



The Overall, the FRA needs to undertake, with the Management Board and possibly usefulness other stakeholders, a thorough review of priorities. The objective should be to of the FRA ensure the available resources are used in the most effective and efficient way,

which may mean a smaller number of projects, stakeholder focus or scope of activities. It will not be possible for the FRA to continue an approach where the Agency tries to fulfil everybody’s expectations to the same extent.

 A strategy for meeting increasing demand for ad-hoc requests should be

developed, in order to ensure that there is a good balance between responding to the external requests and the pertinent needs for research on fundamental rights issues.

 Member States are the duty bearers in the fundamental rights context, and thus

key to reaching a real impact for the rights-holders. The FRA should continue its on-going efforts to be relevant and useful for Member States, in order to create the necessary linkages to deliver pertinent evidence and advice.

 The limits of the mandate of the FRA should be examined and discussed, to

ensure that the Agency’s mandate is sustaining the objective of providing advice and assistance to support the full respect of fundamental rights.

 In particular it should be clarified to what extent the FRA should be mandated to

issue on its own initiative opinions in the legislative process and have a wider mandate to address particular pertinent issues occurring in Member States.



The The FRA should focus on continued consolidation and implementation of the

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Recommendations by the first external evaluation of the European Union’s Fundamental Rights Agency 2012

organisation different management tools developed, such as MATRIX, Quality Management of the FRA System and Performance Measurement Framework. Efforts should be made to ensure that the systems are properly implemented and also used. New initiatives should be avoided.

 The FRA should ensure that staff workload continues to be regularly monitored,

to ensure that there is a reasonable workload.



The working The FRA should continue to strengthen the networking aspect of the Agency’s procedures work, for example by using expert committees and working parties more of the FRA consistently in projects.

 There is a need to put in place procedures/methodologies to respond to ad-hoc

requests.

 There is a need to monitor the development of ad-hoc requests to ensure that

sufficient capacity is available to respond.

In line with Article 31 of the Agency’s Founding Regulation, the Management Board examined the conclusions of the evaluation and issued recommendations to the Commission regarding changes in the

Agency, its working practices and the scope of its mission.

Table 13: Recommendations by the Management Board concerning suggested changes of the FRA’s Founding Regulation (2013)

Recommendations by the Management Board concerning suggested changes of the FRA’s Founding Regulation

 The MB recommends that the founding regulation should be ‘Lisbonised’ in order to

allow the Agency to work in all areas of EU competence, including the area of police and judicial cooperation and cover all rights enshrined in the EU charter of Fundamental Rights.

 The MB therefore recommends amending the founding regulation in order to allow the

Agency to deliver its own motion opinions on proposals for EU legislation that raise fundamental rights issues.

 ‘he MB therefore recommends that a revision of the regulation should establish that

the Multi-Annual Framework is to be adopted by the MB in close consultation with all the three EU Institutions in order to ensure that it takes the respective priorities appropriately into account.

 The MB recommends enlarging the Agency’s tasks enumerated in article 4 of the

founding regulation so they include the possibility for a (group of) Member States to request assistance from the FRA within the scope of its mandate and under the conditions mentioned above.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Recommendations by the Management Board concerning suggested changes of the FRA’s Founding Regulation

 The MB recommends that the Commission should consider amending the founding

regulation in order to include a reference to a possible role for FRA in the framework of Art. 7 TEU (the nuclear option).

 The MB recommends considering tasking the Agency with a continuous,

comprehensive and comparative process of data collection and analysis with regard to the respect for article 2-TEU values.

 The MB recommends that a reference to human rights dialogues with third countries

and possibly other contributions to the work of EU Institutions and the EEAS in external fundamental rights policy matters are inserted in the enumeration of the tasks of the Agency.

 The MB recommends adding a provision to article 4 of the founding regulation

stipulating the Member States shall cooperate with the Agency by facilitating its data collection tasks under this provision.

 The MB recommends that a specific department within the state administration as such

is nominated as National Liaison Authority and out of it an ALO is chosen. This should also apply for the sub-national level.

 The MB recommends that the term of office of MB members replaced every two-and-ahalf

years should end at the same date.

 The MB recommends amending the founding regulation so the quorum for decisions

requiring a two thirds majority of all members changes after three unsuccessful votes to a simple majority of the members present.

 The MB recommends including the EEA in enumeration of organisations with which the

Agency can cooperate under article 8 of the founding regulation.

 The MB recommends that any potential review of the founding regulation should take

due account of the tasks that were assigned to the Agency in recent years, including for instance its role under the UN CRPD monitoring framework, its participation in the Commission’s Roma task force or other examples of standing institutional practice.

Source: FRA, Letter to Viviane Reding including FRA Management Board Recommendations, Vienna, 4 June 2013.

Regarding the usefulness of the FRA, the evaluation of the Agency of 2012 concluded and recommended that the Founding Regulation be ‘Lisbonised’ to enable the Agency to operate in all areas of EU competence, a recommendation which was supported by the Management Board’s response on 4 June 2013 to the European Commission’s Vice-President Viviane Reding on the evaluation’s

recommendations.

As the recommendations proposed by the evaluation and those proposed by the Management Board in

response illustrate, the European Commission would therefore need to propose a reform of the FRA’s

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Founding Regulation, and any revision would then have to be agreed unanimously by the Council and given consent by the European Parliament. However, at the time of writing of this Final Report, a proposal has not been made by the European Commission; nor has a response from the Commission on

these recommendations been received by the Agency.

The Commission had included judicial cooperation on criminal matters in its 2011 proposal for the MAF for 2013–2017, and the European Parliament advocated for this inclusion, but the Council did not agree to this proposal. The 2016 Commission proposal for a Council Decision establishing the MAF for the FRA for 2018–2022 also included ‘police and judicial cooperation’ and ‘social inclusion’. However, the draft decision of the Council opted to restrict the FRA to: ‘judicial cooperation, except in criminal matters’. In the draft recommendation of March 2017, the Rapporteur on the FRA MAF, Mrs Mlinar, was stated to ‘deeply regret the lack of agreement in the Council’ on this matter. On 1 June 2017, the European Parliament gave its consent to the proposal from the Commission, and a decision from the Council is

now awaited in December 2017.

The first independent evaluation of the Agency in 2012 found that the majority of interviewees (including interviewees from international organisations, EU institutions, civil society, Member States and FRA staff) found the FRA mandate as it currently stands to be insufficient to meet the fundamental right needs and therefore for allowing the FRA to be relevant within its current mandate. This was

further supported by the majority of internal and external survey respondents.

With regard to the recommendations concerning the organisation and the working procedures of the Agency, one recommendation made in the 2012 evaluation refers to the prioritisation of FRA’s activities. It was suggested that together with the Management Board and possibly other stakeholders, a thorough review of priorities should be undertaken. In the strategic plan 2013 to 2017, the Mid-Term Review and the annual activity reports, it becomes clear that FRA is still working on numerous different projects and FRA still aims to satisfy the demands of various different stakeholders. FRA has prioritised certain subject areas based on current needs; most recently in the area of migration and asylum. FRA has also increasingly focused on creating synergies between different subject areas in order to more effectively

tackle different priority areas.

In the 2012 evaluation, it was also recommended that FRA shall develop a strategy for meeting increasing demand for ad-hoc requests, in order to ensure that there is a good balance between responding to external requests and the pertinent needs for research on fundamental rights issues. This

recommendation is directly reflected in the Strategic Plan 2013 to 2017 where it was stated that:

FRA will plan and allocate adequate human and financial resources to its advice function. It will assess the needed skills and clarify roles and responsibilities without, however, underestimating the importance of the research and data collection on which the advice is based. The creation of a FRA database of all its opinions, including recommendations contained in reports, formal opinions, and other public statements will allow FRA to keep track of its policy line, and ensure consistency and impact tracking over time.

While no information is available in the annual work programmes or the annual activity reports on the activities that have been undertaken to implement the 2012 recommendation on ad-hoc requests, it can be argued that FRA’s capacity to react to requests has increased. From the planning perspective, in order to implement the evaluation’s recommendation, specific project fiches were created within each area of activity in order to plan and allocate resources to the ad-hoc requests. From the reporting perspective, using the Performance Measurement Framework indicators, the increase in the number of

ad-hoc requests, opinions and so on has been included in the relevant annex on the annual reports.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

In summary, and with regard to the conclusions on the organisation and working procedures of the FRA, the Agency has made considerable progress in improving procedures to monitor and respond to ad-hoc requests, strengthened both the research and networking aspects of the Agency’s work and continued the implementation of different management tools including MATRIX and Performance Measurement Framework. However, there continue to be significant difficulties regarding managing staff workloads and ensuring that staff maintain reasonable workloads. This is also a reflection of the increased demands placed on the Agency to undertake more work in key areas (as discussed below); for example with respect to the asylum crisis, which includes the presence of FRA staff at hotspots in Greece and Italy. In comparison, other Agencies with a mandate to work in the fields of asylum and migration management (in particular Frontex and EASO) have had significant increases in staffing and budget since 2015.

Regarding the recommendations on the usefulness of the FRA, the Agency has accepted the need for changes to its Founding Regulation and done all that it could to enable the recommendations, but the Commission has, in turn, not acted to propose changes to the Agency’s mandate. Interviews, survey responses and stakeholder consultations highlight the continued need for this recommendation to be implemented by the Commission.

3.3.3. The FRA’s strategy

As described in greater detail in other sections of the report, the activities of FRA are based on its Founding Regulation and the thematic areas in which the Agency can carry out these activities are specified in a Multi-Annual Framework covering a five-year period. Since the current Multi-Annual Framework is expiring soon, the definition of the new Multi-Annual Framework (2018 to 2022) is currently in process.

The procedure for developing the FRA strategy started with a consultation of stakeholders for the Multi-Annual Framework 2018 to 2022, which took place in 2015. The consultation involved a large number of civil society organisations and national human rights bodies throughout the EU Member States. The results of the consultation, but also FRA’s experiences gained through projects, ad-hoc requests and interaction with key stakeholders, then resulted in an ‘Opinion of the Management Board of the EU Agency for Fundamental Rights on a new Multi-Annual Framework (2018–2022) for the Agency.’ Based on the Management Board Opinion the Council adopted a draft decision in which the thematic objectives for the period 2018 to 2022 were proposed. In May 2017, the European Parliament agreed to the draft decision of the Council. However, the Parliament regretted the lack of agreement in the Council regarding including police and judicial cooperation in criminal matters in the new Multi

Annual Framework. 69

In regard to the substance, the nine existing MAF thematic areas are largely maintained with only

some changes. This is aligned to the feedback of the stakeholder consultation where around 86% of the

116 respondents wanted all of these areas to be maintained in the future MAF. The slight changes

proposed by the Management Board relate to four issues: add social inclusion of Roma, stress the

relevance of the Charter, mention horizontal obligations and activities and cover criminal matters and

policing. As stated earlier, the latter point especially has not been accepted by the Council.

69 http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP//TEXT+REPORT+A8-2017-0177+0+DOC+XML+V0//EN

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

According to an internal strategy paper, there are five different objectives and related sub-objectives which shall operationalise how FRA is approaching each of the nine thematic areas in the period 2018 to 2022. They are all based on Article 4 of the Founding Regulation which sets out the Agency’s tasks. The

overarching objectives are the following:

 Identifying trends (collecting and analysing comparable data and evidence);

 Contributing to better law making and implementation (providing independent advice);

 Supporting rights-compliant policy responses (providing real-time assistance and expertise);

 Strengthening cooperation with national and local fundamental rights actors (working with

communities of support);

 Effectively promoting rights, values and freedoms.

This list of five objectives consolidates and structures the eight different sub-articles of Article 4 (1) of the Founding Regulation in a simplified way.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

• 4. 

  Evaluation criteria

  This section provides the answers to the evaluation questions. As mentioned in the introduction to this report, these have been grouped according to the evaluation criteria

  Relevance

The assessment of relevance considers the appropriateness of the objectives of an intervention, in relation to the problems it is supposed to address, and the needs of the target group(s). In the context of this evaluation, relevance will thus assess the

extent to which the Agency’s objectives are pertinent to the needs of the FRA target groups, namely the duty bearers (EU

institutions, bodies, offices and agencies and Member States), but will also take rights holders into consideration.

The assessment should also take into consideration the dynamic policy environment in which the Agency is undertaking its work, as this may have an effect on whether the intervention is still relevant to the new policy environment. The assessment of relevance, as well as efficiency assessments, will also feed into the analysis of EU added value and the impacts of the Agency.

Based on the above, two elements within the evaluation questions need to be answered first, namely:

• 1. 

  What are the general, thematic, strategic and operational objectives of the Agency? 2. What are the fundamental rights issues /needs of the FRA target groups?

The Agency’s general, thematic and strategic objectives are developed in section 2.2.3 of this report, and provide the backdrop for the overall assessment.

The fundamental rights needs/problems/issues of the FRA target groups in the EU for the period covered by the evaluation (2013 to 2017), and how these needs changed during this period, was established through desk research and both internal and external surveys, as well as through interviews with three types of stakeholders (see under section 3.1.1 – Needs). This assessment covered the needs of two types of FRA target groups, namely EU Institutions, bodies, offices and agencies and the EU Member States, i.e. the target group to which the Agency provides its assistance according to article 2 of the Founding Regulation, in terms of the type of assistance they need, and on which thematic areas.

4.1.1. To what extent are the Agency’s original objectives still relevant to addressing the

needs, problems and fundamental rights issues within the EU?

The Agency’s main objective, as laid down in Article 2 of the Founding Regulation, is ‘to provide the

relevant institutions, bodies, offices and agencies of the Community and its Member States when implementing Community law with assistance and expertise relating to fundamental rights in order to support them when they take measures or formulate courses of action within their respective spheres

of competence to fully respect fundamental rights’. The Agency’s target groups are thus the duty

bearers, namely EU Institutions, bodies, offices and agencies, as well as the Member States (and in particular the National Human Rights Bodies).

What are the needs of the duty bearers?

As discussed in Section 3.1.1, the Agency’s duty bearers are:

 EU Institutions, bodies, offices and agencies;  EU Member States (as well as candidate countries and potential candidate countries)

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

In terms of thematic areas, the interviewees (both at national and EU level) emphasised the following specific areas of particular fundamental rights needs that were important in the period 2013–2017:

 Judicial and police cooperation;  Discrimination, hate crime, racism, extremism, populism (including the reaction to migrants and

refugees);

 Refugee/migrant crisis;  Social rights;  Roma issues;  Rights of the child;  General fundamental rights decay (e.g. situation in countries like Hungary).

The majority of survey respondents (57%) said that their needs were met by the Agency’s work ‘to a large extent’, while 33% agreed that this was the case to a moderate extent. Overall, the vast majority of survey respondents, 90%, comprising representatives of EU Institutions, National Human Rights Bodies (NHRBs); National Equality Bodies, National Civil Society Organisations, National Liaison Officers (NLOs), National Parliamentary Focal Points and Members of the Fundamental Rights Platform, believed that

their needs were met through their interactions with the Agency or its outputs.

Figure 6: External survey question ‘To what extent are your needs met through these interactions with the Agency or its outputs?’ (n=94)

1%

4% 4%

33%

57%

To a large extent To a moderate extent To a small extent Not at all No Answer

Both external and internal interviewees described their most important fundamental rights needs met by the Agency as relating to FRA data and analysis on fundamental rights issues, as well as opinions and recommendations. One respondent stated that in the past, assistance from the FRA was more related to anti-discrimination policies, but that now more assistance is needed in the field of the

fundamental rights of asylum seekers and migrants.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

With respect to activities of the Agency, the vast majority of interviewees at the EU and the national level discussed the main fundamental rights need the Agency was addressing as the gathering of national-level data across thematic areas to gain a comparative overview across all Member States, as well as its role in the coordination of actors in the fundamental

rights field in the EU. Of lower but still significant importance ‘ FRA events are a breath of fresh air amongst external stakeholders of the Agency is the role of the compared to other fundamental rights Agency, as per the Agency’s mandate, as the sole institutional events ’

representation of fundamental rights at the EU level. Amongst

the Agency’s outputs and events, interviewees and external Regular participant in FRA events

survey respondents focused on Informal communication with the Agency, the Agency’s surveys and other comparative data, online tools, cooperation platforms and thematic groups including the Civil Society and the Fundamental Rights Platform and jointly produced handbooks. In particular, interviewees mentioned the FRA survey on violence against women, FRA opinions and recommendations, contributions to thematic working groups, and regular overviews of migration-related fundamental rights concerns, as addressing their needs.

Is the FRA’s mandate relevant to the duty bearers’ needs?

The mandate

The majority of interviewees (including interviewees from international organisations, EU Institutions, civil society, Member States and FRA staff) found the FRA mandate, as it currently stands, to be insufficient to meet the fundamental right needs and therefore for allowing the FRA to be relevant within its current mandate. This was further supported by the majority of internal and external survey respondents. When discussing the Agency’s mandate, the majority of internal and external survey respondents and interviewees similarly categorised the Agency’s inability to take the initiative of working in specific thematic areas, particularly on judicial and police (including criminal) matters, as a limitation. A survey respondent from the EU Institutions also stated that there was no reason why the FRA MAF should not include police and judicial cooperation in criminal matters. Several stakeholders (incl. Civil society, Scientific Committee members and Academic stakeholders) discussed the inclusion of other prominent potential thematic areas in the next Multi-Annual Framework, particularly social rights in order to ensure the Charter of Fundamental Rights is followed at the EU and the national level, as well as employment rights. Civil society and NHRI members working in existing FRA thematic areas, whilst typically expressing satisfaction with the current tasks the Agency is undertaking, expressed a desire for the Agency to expand its focus within their respective areas. More specific areas that the interviewees mentioned that FRA could cover in addition included: external control on police activities; poverty and some aspects of social exclusion; and the shrinking space for civil society in some Member States. Some interviewees also noticed and welcomed FRA’s recent activities regarding migration and refugee rights, given the increasing importance of the issue within the EU.

In contrast, a minority of interviewees and survey respondents believed the current mandate was sufficient for the Agency to meet the fundamental rights needs, believing that the mandate did not restrict the Agency from working in thematic areas relevant to the fundamental rights needs in the EU. One interviewee at the national level stated that the areas in which FRA cannot undertake any research activities on its own were relatively limited and that, in practice, the Agency could do a lot. Similarly, an interviewee from the EU Institutions stated that the mandate and the objectives of the FRA were relatively broad and not too rigid, leaving room for manoeuvre to research in the areas FRA considered essential, citing the Agency’s actions in the migrant hotspots in 2016 as an example of this flexibility. Interviewees at the national level stated that there was still ‘leverage left for the Agency to do more with its existing mandate’ and that the new Director of the Agency has been successful in this broader

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

interpretation of the mandate, enabling the Agency to act in several new areas and be more able to respond to changing needs in the fundamental rights field of the EU.

Several interviewees stated that there was no problem with the FRA mandate, but rather with its powers to act and its independence to set its own objectives and thematic areas. For example, an

interviewee from the UNHCR stated that it would be useful if the Agency was to be given more leeway within its current mandate to act and would welcome the Agency having greater freedom to issue

opinions and advice in relation to any fundamental rights issues in the EU.

In this regard, several interviewees referred to a tension between the European Commission and the FRA. For example, one interviewee at the national level stated that by having a position on the

Management Board the European Commission was given a disproportionate role in the Founding Regulation, and that as a result the FRA was limited by the fact that the European Commission could

play a significant role in the development of the policy agenda of the FRA. As this is akin to a situation in which a national government official is a board member of a National Human Rights Institution, one

interviewee stated that this would be ‘completely at odds with the independence’ of such an institution.

Similarly, an interviewee from the EU Institutions discussed the structural legal challenge inherent in the fact that the Founding Regulation ensures that the Agency’s

‘ The biggest gap in the mandate is in the former thematic areas are defined by the Commission and the Council 3rd pillar issues – I think they have been quite rather than by the Agency itself, through its Multi-Annual good at stretching discussions and to find a Framework. An interviewee from the Advisory Panel of the way to expand the mandate to work in Fundamental Rights Platform stated that although FRA had Member States. ’ proved to be independent, it had no effective autonomous

Management Board member powers to intervene or to sanction violations of fundamental

rights, which the interviewee believed it should have, but that it was limited to advising the European Commission.

The interviewee stated that in order to strengthen the Agency, it would be beneficial if its opinions could be binding upon the EU Institutions. Another interviewee from the EU Institutions noted that

within the mandate it was given, the FRA had done a very good job in bringing fundamental rights aspects to the policy agenda and making them part of the debate.

As previously discussed, the majority of the interviewees and survey respondents, when discussing the mandate, were of the opinion that the mandate of the FRA was too narrow, needed to be broader and

therefore needed to be changed. The majority of these interviewees mentioned that the mandate of the FRA should be amended to include judicial cooperation and police cooperation in criminal matters. A

survey respondent from the EU Institutions also stated that there was no reason why the FRA MAF should not include police and judicial cooperation in criminal matters. In addition, two interviewees

(from an international organisation and civil society) stated that the FRA should cover social rights and that these should be included in the thematic areas of the FRA.

In summary, there are different perceptions on the Agency mandate amongst the Agency’s diverse stakeholder group. It can be said that those that view FRA as a body responsible for carrying out data

collection and research believe its current mandate is sufficient and changes to the Agency’s remit should be made through the Multi-Annual Framework. However, a significant and diverse group of

stakeholders believe that the current mandate limits the Agency’s ability to meet the fundamental rights needs of the Agency’s target groups, particularly with regard to police and judicial cooperation in

criminal matters. Similarly, those stakeholders who believe the Agency should be doing more than its current main focus on research, believe the Agency’s mandate is insufficient to meet the needs of its

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

target groups and its activities should be expanded, in particular to reflect the need for a fundamental rights watchdog at the EU level, which several respondents identified as a significant role that was

missing.

Were the FRA objectives relevant to the duty bearers’ needs?

As can be seen in Figure 7, a substantial majority of internal stakeholders believe that the Agency’s objectives were, to a large extent, relevant to the needs of the duty bearers in the period 2013–2017. Several staff members highlighted that a multitude of evaluation instruments, including participants’ satisfaction surveys at FRA events and stakeholder consultations (See the 2016 Mid-Term Review of the Strategic Plan 2013–2017, Consultation on Draft Annual Work programmes from 2012 to 2017 and the Programmatic Focus 2018 and 2019, FRA Annual Activity reports, etc.) undertaken by the Agency support this conclusion. One internal stakeholder survey respondent categorised the FRA as an Agency that is becoming increasingly relevant due to changes in the external fundamental rights environment in the EU. As the pressure on Member States authorities, as duty bearers, increases in the field of asylum procedures and return policy, the need to design fundamental rights safeguards and to create evidencebased advice on their implementation has increased the Agency’s relevance for the duty bearers. One internal stakeholder highlighted that ‘the Union has been increasingly challenged to respond to emerging fundamental rights issues, including in the context of the migration situation, global threats, and finally internal challenges to the common values of the EU’. Another respondent focused on the statutory objective of FRA, as expressed in Article 2 of the Founding Regulation, to promote fundamental rights as core to the needs of the duty bearers, who themselves are legally obliged to promote fundamental rights. Staff who believed the Agency’s objectives were, to a moderate extent, relevant to the needs of the duty bearers, highlighted that the limited mandate of the FRA and multiannual planning framework did not allow the Agency to respond to shifting policy needs in the EU over the programming period. Similarly, one respondent stated that the duty bearers’ needs do not always reflect the current challenges that rights holders are facing. Therefore, whilst the Agency’s objective may reflect the duty bearers’ needs, these may be based upon duty bearers’ inaccurate perception of

the rights holders’ needs, and still not reflect the needs of rights holders.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 7: Internal FRA staff survey responses to question ‘To what extent were the Agency’s objectives relevant to the needs of the duty bearers between 2013 and 2017?’ (N=66) This refers to the Agency’s objectives as laid out in the Founding Regulation.

No Answer 3%

Do not know 11%

Not at all 0%

To a small extent 2%

To a moderate extent 20%

To a large extent 65%

0% 10% 20% 30% 40% 50% 60% 70%

National- and EU-level duty bearers answering the stakeholder survey were asked about the extent to which the Agency’s listed thematic areas reflected the areas of their current needs. As can be seen below, the areas that reflect the current needs of the national duty bearers the most (i.e. where over 70% of respondents stated that the thematic area reflected the area of their current needs to a large or moderate extent) are:

 Equality and non-discrimination (77%);  Racism, xenophobia and related intolerance (73%).

The areas which reflect the needs of the national- and EU-level respondents the least (i.e. where over 30% of respondents stated the thematic area reflected the area of their current need to a small extent or not at all) appear to be Judicial cooperation, except in criminal matters.

 Judicial cooperation, except in criminal matters (31%).

However, the rating for ‘judicial cooperation, except in criminal matters’ was qualified by internal and external stakeholder interviewees, who described their significant and increasing fundamental rights needs in the field of police and judicial cooperation including criminal matters – which the Agency was unable to directly address given its restricted mandate. These stakeholders highlighted the fact that the exclusion of criminal matters substantially reduced the relevance of the current judicial cooperation thematic area, and therefore the need for the Agency to work in this area – including criminal matters – was clear. Therefore, we can surmise that, whilst survey respondents didn’t believe judicial cooperation accurately reflected their needs, this was due to the exclusion of judicial cooperation in criminal matters and not due to any decline in the importance of judicial cooperation amongst stakeholders itself.

Despite the Agency’s external stakeholders’ concerns over the FRA’s ability to act in some areas and undertake additional activities, as well as over its independence, which were discussed above, only one

stakeholder contacted – a representative of an inter-governmental organisation with which the Agency

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

works closely – stated that the FRA’s activities were not relevant, due to the fact that the Agency’s objectives were too diluted and, as a result, the Agency did not focus enough on anti-discrimination and

Roma issues. This was despite the fact that the areas of discrimination and Roma integration have consistently formed some of biggest thematic areas by budget throughout the programming period, as expanded upon in section 2.2.3. However, due to the fact that many respondents worked across several thematic areas, it is impossible to qualify these results with the composition of the external survey respondents to understand the level of bias given towards certain thematic areas that respondents work across. In other words, those respondents who work on specific areas – such as ‘Roma’ or ‘antidiscrimination’, for example – will tend to argue that more needs to be done in these areas, even though the Agency may have devoted a considerable proportion of its resources to these areas and less to others. To some extent the evaluation’s results can be said to reflect the inherent bias that many respondents tend to display when asked to assess the significance of certain areas of FRA’s work – with FRA’s stakeholders tending to come from established areas of fundamental rights engagement, and less so from emerging fields that may be increasingly important in years to come (such as information society).

Flexibility of the Agency to address stakeholders’ needs

In order to assess the relevance of the FRA, it is important to investigate whether its objectives, as laid down in the FRA strategic and programming documents, were flexible enough to adapt to changing needs of the Agency’s target groups. In this regard, the majority of FRA staff survey respondents believe that the Agency’s strategic and programming documents are flexible to adequately respond to any changes in the needs of the duty bearers between 2013 and 2017 (see Figure 8).

Figure 8: Internal FRA staff survey responses to question ‘To what extent were the Agency’s strategic and programming documents flexible to adequately respond to any changes in the needs of the duty bearers between 2013 and 2017?’ (N=66)

40.00% 37.88% 34.85%

35.00%

30.00%

25.00%

20.00%

15.00% 12.12% 10.61%

10.00% 4.55%

5.00% 1.52% 1.52%

0.00% To a lage To a To a small Not at all Do not know Not No Answer extent moderate extent applicable, extent there were no change in the needs of the duty bearers between

2013-2017

A substantial majority of internal survey respondents believe that, to a large or to a moderate extent, the Agency’s strategic and programming documents were flexible to adequately respond to the changes

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

in the fundamental rights needs of the duty bearers during the 2013–2017 programming period. Several stakeholders discussed how the Agency’s programming document foresees project priorities and, through their use, the Agency addresses emergent needs. Additionally, the Agency introduced specific instruments for increasing flexibility in thematic areas including projects on ‘additional activities’ in all the Agency’s thematic areas. Similarly, projects in communication, cooperation and awareness raising were developed to meet additional needs. Whilst highlighting an inflexibility in the strategic and programming documents, five interviewees from EU Institutions and NHRIs mentioned as a positive point how the Agency has been able to act in the migrant hotspots despite the constraints of the multiannual

programming.

However, as the most frequent response was ‘to a moderate extent’, improvements could be made to the flexibility of the Agency’s strategic and programming documents. One survey respondent discussed how feedback loops for the Agency to understand changing fundamental rights needs amongst its stakeholders were not substantially developed; a statement which was supported by several interviewees. There is similar disagreement amongst internal stakeholders over the Agency’s balance between multi-annual projects and the limited activities outside of the multi-annual programmes at the discretion of the Agency. Several interviewees and survey respondents discussed a significant challenge for the Agency in reconciling the need to plan multi-annual large projects significantly in advance, with the need to respond to developments at short notice, with one internal stakeholder discussing their belief that the ‘needs of the planning cycle are not always conducive to being responsive to the latest

developments’.

Internal stakeholders highlighted the fact that the multi-annual nature of significant FRA projects required multi-annual planning, which is subject to changes in the external environment and that, due to its commitment to these projects, the Agency does not have the flexible capacity in budget or human resources to respond to emerging issues beyond its existing projects. This is, however, contradicted by the FRA’s flexibility over the past five years and in particular through the Agency’s work at refugee hotspots in Greece and Italy. This was further highlighted by respondents in the external survey, who clearly were of the opinion that the FRA’s objectives were flexible enough to adapt to the changing

needs (see Figure 9).

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Figure 9: FRA external stakeholders survey responses to question ‘To what extent were the Agency’s strategic and programming documents flexible to adequately respond to any changes in the needs of the duty bearers between 2013 and 2017?’ (N=91)

45% 42% 42%

40%

35%

30%

25%

20%

15%

10% 8%

5% 3% 4%

1%

0% To a large extent To a moderate To a small Not at all Do not know No Answer extent extent

The majority of interviewees (including international organisations, EU Institutions, national stakeholders and FRA staff) similarly believed that the Agency has been, to a large or moderate extent, flexible enough to adapt to the changing needs over time. As previously discussed, several interviewees and external and internal survey respondents stated that the FRA had been particularly flexible with regard to the changing fundamental rights needs in the context of the migrant/refugee crisis.

However, a substantial cross-section of these interviewees produced similar suggestions and highlighted a number of fundamental rights trends in the EU that the Agency has not been flexible enough to respond to, including social rights, as well as issues surrounding the rise of populism, extremism and nationalist parties in the EU. One interviewee at the national level also noted that, because the FRA thematic areas were set every five years, these, by design, could not always be as relevant to the current needs, as these are changing throughout the programming period. One survey respondent at the national level stated that ‘the limited FRA mandate and long-term nature of the Agency’s planning and project cycle do not allow [it] to fully respond to shifting policy needs’. However, one interviewee from the Fundamental Rights Platform acknowledged that even though there were specific crises where the FRA was not able to intervene as much, namely the fundamental rights crises in Hungary and Poland, this was due more to limitations of the FRA’s capacity to respond, in terms of available resources, and less to its mandate and objectives. These perceptions demonstrate that some stakeholders do not have a correct understanding of the role of the FRA or the activities it undertakes; indeed, the Agency has responded to human rights crises when relevant to its mandate (in Greece and Hungary for instance) 70

70 FRA, Racism, discrimination, intolerance and extremism: Learning from experiences in Greece and Hungary, December 2013.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Regarding the periodic revisions of the MAF, many internal stakeholders discussed the limitations this places on the Agency in responding to emerging fundamental rights issues. Interviewees from NHRIs and civil society members supported this belief that the multi-annual nature of the MAF hindered the Agency’s relevance in the EU fundamental rights context. Whilst many acknowledged the potential benefit to budgetary management of multi-annual working, as well as the fact that many of the Agency’s projects are only feasible due to this multi-annual approach, they believed there was a strong correlation between the relevance of the Agency and its ability to respond to emerging issues. The significant developments in the European Union in fundamental rights during the programming period, including the so-called migrant crisis, the Syrian civil war, rise in populism across the continent and the recent increase in terror attacks, highlighted the necessity for the Agency to be able to respond to emerging issues as well as the way the Agency was limited both by the exclusion of police and judicial cooperation in criminal matters by the mandate and by the nature of the Multi-Annual Framework required by the Founding Regulation. In contrast, one interviewee stated their opinion that it was not the role of the Agency to respond to every emerging fundamental rights issue, and that the Agency was not designed for a crisis management role. Similarly, it is worth noting that responding to emerging issues is a core competence of the Council of Europe and in particular of its Commissioner for Human Rights, as well as of the OHCHR, and therefore any expansion of the FRA’s role in this regard risks duplicating the work of the Council of Europe. However, as demonstrated by the fact that the Council of Europe was not on the ground in the hotspots in Greece and Italy, there are gaps that remain in responding to these emerging issues. The FRA, despite not being designed for it, filled the void by being present and played an important role in responding to the refugee crisis in the EU.

Additional activities

In terms of the activities under the Agency’s mandate, there was a broader consensus among interviewee and survey respondents that the current activities were to some extent successfully meeting the fundamental rights needs of the EU. One national-level interviewee stated that the objectives of the FRA corresponded with its role, namely to produce data and statistics, and that the Agency was not created to be political. However, several civil society interviewees discussed a desire for an additional role for the Agency as a fundamental rights watchdog in the European Union and involved in all stages of the policy development lifecycle at the EU level.

When asked, just 27% of the of external stakeholder respondents stated that there was a need for the FRA to undertake additional activities.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 10: FRA external surveys response to question ‘In your opinion, is there a need for the Agency to undertake additional activities?’ (N=92)

5%

27%

32%

36%

Yes No Do not know No Answer

Additional activities to be undertaken by the FRA mentioned by the external stakeholder respondents included:

In the field of research:

 More guidance on the use of the Agency’s existing data (survey, raw data) under different thematic

areas;

 More fieldwork and meetings with different stakeholders: e.g. country visits as a regular activity,

especially ‘problematic’ countries which are known for deteriorating respect of human rights.

In the field of cooperation and coordination:

 More advocacy work in conjunction with civil society;  More regular exchange with equality bodies directly, not only with Equinet.  Organise parliamentary meetings on fundamental rights aspects of legislation and on the

application of the Charter of Fundamental Rights.

In the field of dissemination and utility:

 FRA to better disseminate its work;  FRA to increase its presence in the national contexts;  FRA products and opinions to have more weight in the EU;  FRA to ensure its research is taken into account in EU and Member States policy-making;  FRA to offer more training to practitioners and transfer the knowledge and evidence it collects

among its stakeholders across the EU into a more practical way – It is important to know that

training is not part of the Agency’s mandate;

 Increasing its advocacy role, particularly for Member States collecting national-level data.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Business and Fundamental Rights

In recent years, fundamental rights in the business sector has become an often-debated topic among different fundamental rights stakeholders. Often these debates included actors from politics, academia, inter-governmental organisations and civil society, but the business sector itself has not often been given the chance to participate in these discussions. In contrast, FRA aims to actively include industry in its discussions and activities, which has been positively perceived by those stakeholders. One

interviewee mentioned:

Many fundamental rights actors from civil society and academia do not want to engage with industry, even if their activities have direct implications for fundamental rights. It is therefore very positive that FRA actively tries to establish a dialogue with the business sector and in this way,

attempts to bridge the gap between policy-makers/civil society/academia and businesses.

Examples of FRA’s strategy to include businesses in the discussions relate to the fundamental rights platform which is not only open to civil society but also to businesses (mostly represented by law firms). Furthermore, during the Fundamental Rights Forum in 2016 a panel was held on the digital age and privacy where both ‘traditional fundamental rights stakeholders’ as well as speakers from big datadriven companies participated. Also, during the 2018 Fundamental Rights Forum FRA intends to open

the dialogue with the tech industry and in addition aims to include a panel on sustainable business.

Apart from these engagement activities, FRA has also in its research aimed to include the views of businesses. For example, in research on severe forms of labour exploitation and research on the impact

of the migration crisis on local businesses/tourism FRA collects the views of local businesses.

Recently, the Council of the EU requested a legal opinion of FRA in its Conclusions on business and human rights. The opinion sought from FRA was to look at ‘possible avenues to lower barriers for access to remedy at the EU level’. The Opinion is over 80 pages long and includes 21 recommendations in regard to access to remedy. The Opinion was well received by the Council and by the European Commission, and currently discussions are being held on how FRA can conduct follow-up research in this

field.

The examples highlighted above show that FRA interprets Article 10 of the Founding Regulation in a broad manner. Furthermore, the examples also illustrate that FRA’s duty bearers are expecting FRA to

be active in this field. 71 The Article stipulates that ‘[t]he Agency shall closely cooperate with nongovernmental

 organisations and with institutions of civil society, active in the field of fundamental rights’, which includes stakeholders such as businesses. Thus, FRA rightly acknowledges the growing impact that certain businesses have on different fundamental rights. Examples include the impact of bigdata-driven companies on the right to private life or the link between extreme forms of labour exploitation in particular industries such as agriculture. At the same time, FRA has begun to explore the power of the business sector to raise awareness of fundamental rights and related FRA outputs. In a strategy paper FRA mentioned that engagement with the business sector should also aim at publicising fundamental rights concerns and FRA findings. For instance, product packaging could be used as

medium to inform the public about issues like domestic violence.

To conclude, FRA’s interaction with industry and related activities demonstrates that the Agency interprets its mandate holistically and that it reacts in a flexible manner to emerging needs and

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

opportunities.

4.1.2. Have the recommendations on the relevance from the previous evaluation been implemented by the Agency?

Throughout the 2013–2017 period, there were conflicting expectations and aspirations in relation to the Agency’s mandate, amongst external stakeholders of the Agency. As stated in the FRA Internal Memo of 23 September 2014, the Agency ‘has a wide mandate, but some changes to the Founding Regulation appear needed and timely’. As the Agency’s Founding Regulation was adopted more than two years before the Treaty of Lisbon entered into force, it still refers to the Agency meeting its objective within the ‘competencies of the Community as laid down in the Treaty establishing the European Community’. The Internal Memo lays out the Agency’s belief that this refers to ‘the competencies of the EU as they develop’. As expanded upon in Section 2.1, the Treaty of Lisbon significantly increased the EU competences, particularly in the sphere of police and judicial cooperation. Following this interpretation, the Agency believes it can therefore deal with all areas covered by an EU competence, including the expanded competences after the introduction of the Treaty of Lisbon. As expressed in the Opinion of the Management Board of the Agency on a new Multi-Annual Framework (2018–2022) for the Agency, the Management Board supports the recommendation to remove the exclusion of ‘criminal matters’ from the MAF area ‘judicial cooperation’, an opinion that has also been ‘repeatedly stressed’ by the European Parliament. As Article 4 (1 (d)) of the Agency’s Founding Regulation stipulates, the Agency shall ‘formulate and publish conclusions and opinions on specific thematic topics, for the Union institutions’, and given the fact that the Agency receives requests from the European Parliament on police and judicial cooperation in criminal matters, this has resulted in confusion both within the Agency and amongst Parliament representatives.

Article 3 (5) of the FRA Founding Regulation states that ‘the Agency shall carry out its tasks within the thematic areas determined by the Multi-Annual Framework’. Although the FRA Multi-Annual Framework (MAF) for 2013–2017 covers access to justice, victims of crime and judicial cooperation except in criminal matters, the MAF does not include police and judicial cooperation in criminal matters or social rights/ inclusion.

In light of the above, the first independent external evaluation of the Agency of 2012 concluded and

recommended 72 that the Founding Regulation be ‘Lisbonised’ to enable the Agency to operate in all

areas of EU competence, a recommendation which was supported by the Management Board’s response on 4 June 2013 to the European Commission’s Vice-President Viviane Reding on the evaluation’s recommendations. A response from the Commission on this recommendation was not received by the Agency. However, whilst the Agency is unable to operate in all aspects of fundamental rights in the European Union, the situation within the Union has changed considerably during the programming period due, in part, to the rise of the migrant crisis in 2015, anti-immigration sentiment across the EU and concerns surrounding the rights of terror suspects. These changes have raised the importance of police and judicial cooperation, including criminal matters, and therefore led to a subsequent reduction in the relevance of the Agency.

72 FRA, Evaluation of the European Union Agency for Fundamental Rights, Final Report, November 2012.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Table 14: Recommendations by the Management Board concerning suggested changes of the FRA’s Founding Regulation (2013)

Recommendations by the Management Board concerning suggested changes of the FRA’s Founding Regulation

1 The Founding Regulation be ‘Lisbonised’, enabling the Agency to work in all areas of the EU competence including the area of police and judicial cooperation and cover all rights enshrined in the EU Charter of Fundamental Rights, which should be clearly established as the normative backbone of the Agency’s work.

2 The Founding Regulation be amended in order to allow the Agency to develop and deliver its own opinions on proposals for EU legislation that raise fundamental rights issues based on its expertise, data collection and analysis.

3 A revision of the Founding Regulation establishing that the Multi-Annual Framework be adopted by the Management board after close consultation with the EU Institutions.

4 Enlarging the Agency’s tasks enumerated in article 4 of the Founding Regulation so they can include the possibility for a (group of) Member States to request assistance from the FRA within the scope of its mandate and under the conditions mentioned above.

5 The Founding Regulation to be amended in order to include a reference to a possible role for the FRA in the framework of art. 7 TEU.

Source: FRA, Letter to Viviane Reding including FRA Management Board Recommendations, Vienna, 4 June 2013.

In order to reflect the recommendations proposed by the Management Board, and as Table 14 illustrates, the European Commission would therefore need to propose a reform of the FRA’s Founding Regulation, and any revision would then have to be agreed unanimously by the Council and given consent by the European Parliament. However, at the time of writing of this Final Report, a proposal has

not been made by the European Commission.

The Commission had included judicial cooperation on criminal matters in its 2011 proposal for the MAF for 2013–2017, and the European Parliament advocated for this inclusion, but the Council did not agree to this proposal. 73 The 2016 Commission proposal for a Council Decision establishing the MAF for the FRA for 2018–2022 also included ‘police and judicial cooperation’ and ‘social inclusion’. 74 However, the draft decision of the Council did not include the thematic areas of police cooperation and judicial

cooperation. 75 In the draft recommendation of March 2017, the Rapporteur on the FRA MAF, Mrs Mlinar, was stated to ‘deeply regret the lack of agreement in the Council’ on this matter. 76 On 1 June

2017, the European Parliament gave its consent to the proposal from the Commission, and a decision from the Council is now awaited.

73 http://www.europarl.europa.eu/sides/getDoc.do?type=REPORT&reference=A7-2012-0361&language=EN.

74 http://ec.europa.eu/justice/fundamental-rights/files/fra_ma_fw_en.pdf

75 http://data.consilium.europa.eu/doc/document/ST-14423-2016-INIT/en/pdf

76 http://www.europarl.europa.eu/sides/getDoc.do?type=COMPARL&reference=PE-601.223&format=PDF&language=EN&secondRef=01

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

To conclude, the Agency’s internal and external stakeholders believe the FRA’s objectives are relevant to the needs of duty bearers, and have been throughout the programming period. However, a substantial cross-section of the Agency’s stakeholders are of the opinion that the FRA’s remit should be changed in order for the Agency to meet an existing fundamental rights need in the EU around police and judicial cooperation in criminal matters. This was supported by interviews with international, EU and nationallevel stakeholders, and FRA staff; stakeholder consultations undertaken by the Agency; and internal and external stakeholder surveys, as well as substantial desk research. Several interviewees were of the opinion that the mandate should be changed to include police and judicial cooperation in criminal matters as well as social inclusion in the MAF, whereas others had wider concerns about the powers of

the FRA to set its agenda and to act, and the position of the European Commission herein.

Almost 40% of national-level respondents stated that there was a need for the FRA to undertake new activities; however, there was significant variation amongst stakeholder groups perceptions on what

these activities should be.

The Agency has implemented the recommendations on the relevance from the previous evaluation that fell within its purview for action. However, the European Commission has not responded to the Agency’s recommendations regarding the implementation of recommendations on the Agency’s mandate and

objectives from the previous evaluation.

In terms of the thematic areas of the FRA, the majority of the duty bearers at EU and national level stated that the thematic areas ‘Racism, xenophobia and related intolerance’ and ‘Immigration and integration of migrants, visa and border control’ reflect the areas of their current needs. The majority of respondents identified the thematic area ‘judicial cooperation’ and ‘Roma integration’ (for national-level respondents) and ‘Rights of the child’ for EU-level respondents, among others, as thematic areas that reflect their needs to a small extent, or did not reflect their needs at all. However, as previously discussed, an implicit bias exists in the responses of stakeholders on the relevance of topic areas with regard to the thematic areas the stakeholders themselves operate in. Additionally, the low rating of judicial cooperation reflects the Agency’s limitations within that thematic area, and not the lack of

importance of the thematic area itself.

The majority of FRA staff and external respondents at EU and national level are of the opinion that the Agency’s strategic and programming documents were flexible to adequately respond to any changes in

the needs.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Effectiveness, Utility and Impact

Impact is a crucial criterion in evaluations for understanding the level of (long-term) effect of an intervention. Furthermore, the impact of the Agency will need to be assessed in order to assess its effectiveness and utility. It is important to clearly define ‘impact’ and distinguish it from the other fo rms of effects.

The assessment of impacts should consider the effects of the Agency on the wider context, be they direct (i.e. the logical and intended consequence of the activities undertaken by the Agency) or indirect (i.e. through some form of multiplier). Measuring and quantifying impacts is a requirement of this evaluation. However, evaluating the impact of interventions is intrinsically difficult as impacts are usually realised in the longer term, and so it is not easy to determine whether the evidenced developments were influenced by the intervention in question and/or rival explanations, such as political actions at the national/international level. Hence, and based on the monitoring data collected by the Agency, for the purpose of the evaluation, impact will examine the extent to which the Agency’s work and outputs have contributed to the aspirational impacts, directly or indirectly. As su ch the chosen indicators provide a mixture of short- and long-term achievements going beyond the scope of the data collected by the Agency.

Utility is the assessment of the correlation between the effects of the Agency and the identified needs of the stakeholders and to what degree the satisfaction differs according to the different stakeholder groups.

Finally, effectiveness should consider the degree of achievement of the intervention’s various objectives.

It has to be noted that the 2012 evaluation concluded that the FRA was effective in terms of responding to stakeholders’ data and information needs; however, there is a need to review the Agency’s priorities to ensure the effective use of resources.

In terms of utility, it is important to assess stakeholder perceptions of the usefulness, uptake and accessibility of FRA’s outputs and whether these are perceived to be meeting their needs, as well as addressing the wider fundamental rights issues. In assessing effectiveness, it is important to examine both the internal and external effectiveness of the Agency. In terms of the Agency’s internal effectiveness, it is necessary to examine the internal ways of working in terms of day -to-day operations, resourcing and the production of outputs. It is also important to examine the associated internal policies and procedures in order

to fairly assess the Agency’s internal effectiveness. It is also important to understand how staff respond to these internal

processes.

4.2.1. Impact

This section sets out to answer the following evaluation questions:

 EQ4 – How successful has FRA been in achieving the expected effects (outputs, results, impacts), in

light of its objectives, mandate and tasks, as defined in its Founding Regulation?

 EQ10 – To what extent does the impact achieved by the FRA’s activities correspond to and meet

existing stakeholders’ needs?

 EQ11 – Are there any additional outputs/results that were not foreseen initially in the multi-annual

and annual work programmes?

 EQ8 – To what extent have FRA activities had an impact on EU policy and practices in MS/third

countries/NHRBs/the fundamental rights field?

A number of the issues to be covered here overlap with other issues presented in the different sections of the report. As such, this section will focus on the impact of the Agency. The first step when assessing impact is to develop clear definitions. Given the specific format of the FRA, its mandate and

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

independence, impact should be demonstrated beyond what would usually be done for other decentralised agencies.

In order to address the question of impact, the first step is to clearly set out the mandate of the Agency as well as the Paris Principles, or at least those which are relevant to the FRA. Second, given the complexity of the matter, a typology of impacts should be assessed. The typology should at least include

the following elements:

 Direct v. indirect impact – as discussed elsewhere, the main target group of the FRA are duty

bearers; however (and in line with the Paris Principle), the essence of the role of the FRA is ultimately to have an impact on rights holders;

 Impact at EU level v. impact at national level – one of the main conclusions from the first

evaluation of the Agency was that impact could not be demonstrated at national level.

 The visibility of impacts – there is a slight dichotomy between the extent of the impact of the

Agency and the visibility of these impacts. As an example, good working relationships between FRA and Commission staff might not be very visible, but can have important impacts in ensuring the

Commission proposes legislation which is in line with the Union’s Fundamental Rights standards;

 Measurable v. intangible impacts – while some of the impacts of the Agency are measurable, other

impacts are not (the impacts of a country visit might not be measurable – see below).

When questioning stakeholders about the impact of the Agency, one key element is striking; there is not always a common understanding of what the objectives of the Agency are. This lack of common understanding can lead to situations where some stakeholders have a more negative view of the Agency’s impact given they expect its objectives to be much wider than they actually are according to its

Founding Regulation, particularly in regard to the difference between national and EU competences.

As an example, one civil society representative highlighted that in his view, ‘most of the population have little idea what fundamental rights they have and if these are being infringed, which is a failure of the Agency’. This statement, although it is an outlier, does provide some insight on how the Agency is perceived by some stakeholders, given that raising awareness of fundamental rights amongst the general population is not a direct objective of the Agency. The competency of the FRA is limited to act where the EU has competence. This can lead to frustration when the distinction between EU and national competence is not understood. Following the principle of subsidiarity, the fundamental rights architecture in the EU includes FRA (acting on issues of EU competence) and NHRIs, Ombuds and

Equality Bodies (acting on issues of national competence) at the national level.

Art. 4 of the Founding Regulation states that one of the Tasks of the Agency is to ‘develop a communication strategy and promote dialogue with civil society, in order to raise public awareness of fundamental rights and actively disseminate information about its work’, again placing the Agency in a support role to help duty bearers address the needs of rights holders. This statement highlights the care with which the data relating to impact has to be interrogated. The Management Board recently agreed on launching the Agency’s first EU-wide survey on the general population’s understanding and

experience of fundamental rights in practice.

At the core of the Agency’s evaluation, and on the basis of the conclusions and recommendations for the first independent evaluation of the FRA, are the impacts that the Agency has. Impacts are the highest level of effects and consequently the most difficult to measure and appropriate to specific activities. In order to create the causal link between the activities and subsequent outputs and results of

the Agency with the impacts it might have, a few examples have been developed. One of the 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

specificities of the Agency is its dual role in carrying out primary and original research and informing policy-making. This dual aspect is key to the achievement of certain impacts. Given the mandate of the Agency, most of the impacts it achieves are indirect. A key aspect of the impact the Agency has is

reflected in its objective to provide ‘assistance and expertise’ through specific tasks such as research,

data collection, conclusions and opinions and using this to support and influence policy-making at the EU and national level. This is explained in detail in the case studies (see Appendices). Another example demonstrating how the Agency’s outputs is translated into impacts is the creation of the working party

on improving hate crime reporting and recording.

77

Figure 11: How the FRA generates impact – the case of the Working Party on improving reporting and recording of hate crime

Example of the chain of impact – the Working Party on improving reporting and recording of hate crime

At the inception of this project research was undertaken by the Agency (1) . On the basis of research and publications undertaken by the Agency, including the EU-MIDIS survey, a special survey on Jewish

people’s experiences and perceptions of discrimination and hate crime and a publication entitled ‘Making hate crime visible in the European Union: acknowledging victims’ rights’, a conference was

organised by the FRA in cooperation with the Lithuanian Presidency of the Council of the EU in November 2013 (2) . 78 The conclusion of the conference set out a number of practical recommendations,

including the need to ‘facilitate good practice exchange and assist the Member States in their effort to

77 See: http://fra.europa.eu/en/project/2015/working-party-improving-reporting-and-recording-hate-crime-eu

78 Combating hate crime in the EU - Giving victims a face and a voice, 12-13 November 2013.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

develop effective methods to encourage reporting and ensure appropriate and adequate recording of

hate crimes’ (3) 79 . Interestingly, at the time, the majority of participants in the Fundamental Rights

Conference in November 2013 reported that they expected it to influence policy-making to a high or very high extent (53.8% of respondents). 80

In turn, the conference influenced the conclusions of the subsequent Justice and Home Affairs Council meeting which took place in December of the same year. The Council Conclusions, referring to the research undertaken by the Agency, and using the same language as that used for the conference, called upon the FRA to ‘to work together with Member States to facilitate exchange of good practices and assist the Member States at their request in their effort to develop effective methods to encourage reporting and ensure proper recording of hate crimes’. 81 (4)

In November 2014, the first meeting of the Working Party on improving reporting and recording of hate crime in the EU was held in Rome with 25 Member States, the European Commission and the OSCE’S Office for Democratic Institutions and Human Rights (5) . The Working Group has developed a compendium of practices for combating hate crime. Furthermore, as a result of its work, the Agency has been invited by the Commission to join the High-Level Group to combat racism, xenophobia and other forms of intolerance (6) and facilitate its Subgroup on improving recording and data collection on hate crime.

This example showcases how the Agency’s ability to undertake research, communicate and disseminate it and inform policy-making is a key element in ensuring that it can achieve as high a level of impact as possible.

Impact related to data collection and research

As demonstrated in the example above, and highlighted by the overwhelming majority of stakeholders consulted, the data collection and research activities undertaken by the Agency are among its most

important activities.

The quality of the Agency’s research outputs is undisputed by stakeholders consulted, regardless of their position and the organisation they represent. The Agency’s research is praised for its quality, and (as discussed in relation to coordination and coherence as well as added value) the fact that it provides data which is comparable across the EU and over time, as well as being scientific and objective as discussed in section 4.3.1. In addition, the independence of the Agency ensures that the data collected and research undertaken is seen as unbiased. Reports and publications are a key output of the Agency; in 2013, FRA produced 29 publications, in 2014 it produced 53 and in 2015 it produced 32. For publications, FRA was the highest-ranking EU agency in terms of EU Bookshop orders in 2016. In 2016, FRA published its research findings in 306 publications, including reports, papers, handbooks, online publications, factsheets etc., which its stakeholders received well and used widely. Overall, the Agency

disseminated some 73,419 print publications to interested stakeholders in 2016. 82 Survey results are the

79 Point 3.6, Conference conclusions, Combating hate crime in the EU – Giving victims a face and a voice, 12–13 November 2013.

80 To what extent do you expect this conference to influence policy making at the national and/or EU level? N= 147, in FRC Evaluation results

2013.

81 Council Conclusions on combating hate crime in the European Union, Justice and Home Affairs Council meeting, Brussels, 5 and 6 December

2013.

82 Annual Activity Reports, 2013, 2014, 2015 and 2016.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Agency’s most used output (61% of external survey respondents using them ‘often’ or ‘very often’), followed by newsletters (56%) and thematic reports (52%).

In terms of the impact this research and these publications have, a few examples are provided below. One of the key anticipated impacts of the Violence Against Women survey was the provision of necessary evidence to develop policies and legislation to address violence against women. 83 In 2017, the Council decided upon the European Union’s accession to the Council of Europe Convention on preventing and combating violence against women and domestic violence (also known as the Istanbul Convention). The Violence Against Women survey was a key input into the Council’s decision, as demonstrated by numerous mentions of the survey in official documents – most importantly, the Proposal for a Council Decision on the conclusion, by the European Union, of the Council of Europe Convention on preventing and combating violence against women and domestic violence 84 , which directly quotes the results of the survey as highlighting the high level of violence against women in the

EU.

Case Study – Violence Against Women Survey

Violence against women is a key human rights problem that impacts all EU Member States, affecting and undermining core fundamental rights such as dignity, access to justice and gender equality (amongst others). In this context, the FRA launched, in its Annual Work Programme 2010, the project ‘Violence against women: an EU-wide survey’, aiming to provide the EU and the Member States, for the first time, with comparable data which can be used to develop policies to combat violence against women, protect the victims, and raise awareness on the issue, both among the general public and among specific groups of practitioners working with victims of violence.

The survey filled a gap identified by European Institutions and Member States of a persistent lack of comparable data at EU level that decision-makers need in order to shape informed, targeted policies. For the first time, comparable data on women’s experiences of violence was made available to policymakers and practitioners in all EU Member States, particularly for the development of policies and other measures to combat violence against women, and also enabling comparisons of the extent of violence

against women in different Member States. 85

Short-term impacts

The results of the survey were presented in 44 events in 2014, 12 in 2015 and 16 in 2016, including international conferences and seminars, working groups (e.g. Eurostat working group on crime statistics, Council of Europe Committee (GREVIO) responsible for monitoring implementation of the Istanbul Convention, Council of Europe drafting group on female genital mutilation and forced marriage, UNFPA technical advisory committee on strengthening regional and national capacities for measuring violence against women, German Bundestag hearing on the survey’s results, UK Parliament’s Joint Committee on

83 ICF, FRA Internal Monitoring and Evaluation Contract Specific Request 3 Work Package 4: ex-ante and ex-post evaluations 25 November 2014 -

Ex-post evaluation report: FRA Survey on Violence against women (VAW).

84 COM/2016/0109 final.

85 FRA website, http://fra.europa.eu/en/project/2012/fra-survey-gender-based-violence-against-women (accessed on 2 October 2017)

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Human Rights (JCHR)), as well as several roundtables in other EU Member States.

The launch of survey results was the biggest launch of a FRA product in the history of the Agency, with at least 1,500 news articles and broadcast reportages during the first four days across EU member States and the world. Media statistics show that media in all EU Member States reported on the survey and, on the launch day, all EU media with the widest reach covered the report in their headline news. 86 Moreover, within the first seven days FRA’s website registered very high traffic, with around 12,000 publication page views and over 8,000 downloads of the main results report. 87

Data on downloads of the Violence Against Women Survey outputs also shows that the results have reached a large number of stakeholders. The main products downloaded from the FRA website in 2014 were the main survey results (32,598 downloads) and the factsheet (26,221 downloads), with clear evidence of a long-term interest of stakeholders in the survey results, particularly in the case of the main results report, which was downloaded almost 32,000 times even in 2016.

Long-term impacts

Research findings from the Violence Against Women Survey have been frequently used by EU and national-level policy makers and practitioners. Stakeholders reported that the FRA, through its Violence Against Women Survey, had an impact on the development of EU legislation, policies and practices and that the survey results were a driving factor in the EU signing the Council of Europe Convention on preventing and combating violence against women and domestic violence (Istanbul Convention) in 2017. The Istanbul Convention, presented in 2011 and entered into force in August 2014, was the first European Convention on violence against women, including domestic violence, and is also legally binding. It sets out minimum standards for Member States signing and ratifying the Convention on the prevention of violence against women, the protection of victims and the prosecution of perpetrators, and calls for signatories to take the appropriate legislative steps to this end. 88

Finally, the Survey contributed to providing Member States with a well-documented methodology on how to collect data on violence against women. Furthermore, the survey has developed a set of comparable indicators at European and national level which can be used as a basis for the development of national statistical instruments.

The survey questionnaire is being replicated in 10 non-EU Member States by the Office for Security and Cooperation in Europe (OSCE). FRA is part of the steering committee for the survey. The survey forms a basis for the development of a survey on violence against women and men, which Eurostat is piloting in selected EU Member States. FRA is part of the expert group at Eurostat addressing this.

Other types of outputs are also seen in a very positive light and appear to be used to a high degree, in particular the legal handbooks the Agency produces in conjunction with the Council of Europe, given

86 The findings appeared on the front page of several EU newspapers and were included in the morning and evening news of national radio and

television stations across the EU. Amongst the countries where most coverage was recorded, the highest impact was in Germany

with 58 media reporting on the survey, followed by Spain with 38 media and Italy with 26.

87 FRA Media analysis – VAW report and survey 2014

88 European Parliament (2016), The Issue of Violence Against Women in the European Union, available at

http://www.europarl.europa.eu/RegData/etudes/STUD/2016/556931/IPOL_STU(2016)556931_EN.pdf 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

their reach (by being translated in all EU languages) and practicality. These handbooks provide hands-on guidance on legal principles in the areas of non-discrimination, data protection, asylum and immigration, children’s rights, and access to justice. They provide an example of the direct impact the Agency has. The impact of these handbooks is illustrated in the ex-post evaluation of the Handbook on

data protection law, where all respondents to a survey undertaken found the publication to be ‘useful’ of ‘very useful’. 89 The handbooks are published in all EU languages: almost 100,000 copies had been

disseminated by the end of 2016, while around 340,000 had been downloaded by mid-2016. 90

The importance of the data collected by the Agency, and in particular the fact that it is comparable across Member States and independent, was highlighted by a number of stakeholders consulted,

regardless of the area they are involved in or the organisation they represent or work for. In the words of a stakeholder from the Council of Europe, ‘evidence-based human rights policy can only occur when

someone collects the information and the Agency is the only one in Europe that does so’.

Finally, and linked to the issue of communication (see section

‘ When I am drafting policy 4.2.4), one aspect which was mentioned a number of times recommendations at the national level was the way in which the Agency capitalises on the research it

and when I am lobbying policy-makers, undertakes. The FRA has been recognised as good at using the outputs are a reliable source of existing research in different forms of publication and

information which gives weight to my dissemination. The creation of the Fundamental Rights recommendation. ’ Promotion Department partly stemmed from this desire to

National Civil Society representative make better use of existing research. While the Agency

consistently and increasingly uses this data to inform the opinions it provides (such as the ad-hoc legal opinions requested by the European Parliament), this was seen by a small number of interviewees as being hampered by a lack of consistent and institutionalised links with Member States. While the Agency’s increasing work with national parliaments and through National Liaison Officers is recognised by stakeholders, the positive impact of having an FRA member of staff knowing the local context and culture of a specific Member State was seen by a number of interviewees with a deep understanding of the Agency as crucial to better take the ‘pulse’ of the country and be aware of when to highlight or publish reports in specific areas.

FRA provides assistance and expert opinions on fundamental rights to its stakeholders and to its networks, at national, EU and international level. FRA develops its stand-alone legal or socio-legal

published ‘Opinions’ on the basis of requests received from key institutions; namely, the European Parliament, Council or Commission. FRA can also issue opinions and conclusions to EU Institutions and Member States when they develop and implement EU legislation. Typically, FRA’s expert assistance is

delivered through reports on an on-going basis, requests made by an EU Institution or by a Member State, training materials and programmes, and by sharing ‘promising practices’.

While some stakeholders saw the value of the Agency as lying more in the research it undertakes than in the opinions it issues, as noted above, others highlighted the importance of the Agency’s multidisciplinary

approach. According to a high-level representative of Frontex, ‘FRA is really focusing on the real problems with a clear aim to improve the situation and not just criticise’.

89 ICF, FRA Internal Monitoring and Evaluation Contract Specific Request 3 Work Package 4: ex-ante and ex-post evaluations 12 November 2014 –

Ex-post evaluation report: Handbook on European data protection case law.

90 Annual Activity Report 2016.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Legal opinions

According to article 4(1)(d) of the Founding Regulation, the Agency shall ‘formulate and publish conclusions and opinions on specific thematic topics, for the Union institutions and the Member States when implementing Community law, either on its own initiative or at the request of the European Parliament, the Council or the Commission’. In accordance with Article 4(2) of the Founding Regulation FRA opinions can only concern EU legislative proposals when requested by the European Parliament, the Council of the European Union or the European Commission. In practice, these legal opinions are often drafted at the request of the European Parliament; less so at the request of the European Commission or the Council.

While the value of the research undertaken by the Agency is undisputed, a few stakeholders, mainly from the European Commission, highlighted that the unique value they found was in the research itself and the multi-disciplinary approach of the Agency. For instance, one Commission official pointed out that as a large administration, the Commission had a lot of in-house lawyers, but what it lacked was other social scientist and economists. He suggested that the value of the Agency’s legal opinions was limited, stating that ‘the FRA’s legal opinions do not have the same gravitas as there are higher authorities than the Agency in this matter. The FRA is the leading provider of surveys in Europe if not the world, they should focus on their strengths.’ This was backed by the representative of an NHRI, who stated that they more often used the findings of specific pieces of research to draw their own legal conclusions rather than using the Agency’s own legal conclusions. On the other hand, MEPs had a positive view of the Agency’s legal opinions, reflected by the increase in the number of opinions requested by the Parliament since 2016 as well as responses to the interviews conducted. Further evidence of impact has been identified in academic research; research undertaken highlighted that FRA and EDPS legal opinions “have often found their way to the

legislative process through the EP positions, bringing ‘ We are very satisfied with the data considerable added value to the legal instruments provided by the FRA; it is both relevant

91

concerned’. and of high quality. More importantly they are trustworthy as the Agency does

The significant role of the Agency’s legal opinions on specific not have an agenda to defend. ’

legislative has been noted. For instance, the recent legal European Commission Official

opinion on the impact on fundamental rights of the proposed Regulation on the European Travel Information and Authorisation System (ETIAS) had a big impact on the way in which the Parliament has amended the draft legislation. In some cases, the role of the FRA is important in areas where no fundamental rights impact assessment has been carried out, such as in the case of the Directive on combating Terrorism.

Other activities

The visibility of the Agency and the impact it can have are not always correlated. A large number of stakeholders in the EU Institutions as well as FRA staff have highlighted the importance of the unofficial links between the Agency and other EU Institutions, bodies, offices and agencies. Stakeholders in the European Commission mentioned the usefulness (and ultimate impact) of phone calls with their colleagues at the FRA when developing legislation with the need to take into account fundamental rights

91 Fyhr, Kim, Making Fundamental Rights a Reality in EU Legislative Process : Ex ante Review of Proposals for EU Legislative Measures for their

Compatibility with the Charter of Fundamental Rights of the European Union, p. 224 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

principles. These interactions are not visible to the outside world but arguably have a large impact. This view was also backed by different stakeholder groups (MEPs, and international organisations).

A few representatives of civil society organisations highlighted how the EU Institutions regularly take positions that directly contradict the Agency’s recommendations and that these recommendations are routinely ignored at both the EU and national level. One international organisation felt this was not due to any lack of quality in the advice given by the Agency but rather because the political context is such that ‘professional research and well-meaning advice presented to EU policy and decision makers in some cases is irrelevant’. This argument was offset by some NHRIs, who stated that they could make use of the data and research undertaken by the Agency in order to back up their findings and conclusions.

However, one interviewee from the secretariat of the Council of the European Union discussed how the Agency had been ‘quite effective in bringing the findings from its activities to attention’, noting that the effort was significant and substantial. This was supported by an interviewee from an international organisation who noted that the Agency had buy-in from policy-makers in certain areas, particularly in disabilities and other politically less sensitive areas, but not in other areas.

Interestingly, the majority of interviewees from the Commission or EU agencies highlighted the good working relationships they had with their FRA colleagues, and the positive role this plays in the development of fundamental-rights compliant legislation. While this is not the most visible of impacts, it is arguably one of the Agency’s most important.

Regarding the extent to which the FRA’s activities have an impact on EU policy and practices in Member States, third countries, NHRIs and the fundamental rights field, a distinction needs to be made between any direct impact the Agency has and the indirect impact of the Agency’s activities. The majority of respondents felt that it was

almost impossible to quantify the Agency’s direct impact and that ‘ At the moment FRA is not very visible in every EU country. Their research and the

instead, its impact focused on the indirect use of its activities or work they do is excellent, but not

outputs by external stakeholders in the Member States. One everyone knows about their work. ’

Management Board member discussed their belief that the Agency

has had little direct impact on policy development but that the Member of the LIBE committee

Agency had impacted LGBTI, domestic violence and children’s rights issues in terms of raising awareness and enabling the comparability of information to highlight where pitfalls exist. Linked to these indirect impacts, several national-level external stakeholders discussed the use of the Agency’s outputs that fed into the development of national policy and practices. Additionally, one national-level stakeholder highlighted how the use of the Agency’s handbooks had been important in the development of national practices for law enforcement in that Member State. A National Liaison Officer described the use of the Agency’s research activities, in particular the Violence Against Women survey, by national policy-makers to inform the national-level discussion of these issues.

While most of the impacts the Agency has at national level are indirect, through cooperation with local partners or the use that local actors make of the FRA’s research, country visits play an interesting and different role. Country visits are part of the Agency’s objective to provide assistance and expertise relating to fundamental rights to Member States. Usually taking place over two days and focusing on fundamental right issues specific to the country, these visits by the Director are undertaken at the political level. One Finnish stakeholder spoke very highly of the visit undertaken by the Director in Finland. Non-negligible results included the sharing of best practice on surveillance of legislation which postponed on-going discussion in the country to ensure a proposed local legislative initiative is fundamental rights compliant. Beyond this, it was reported that high-level political officials were very

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

positively influenced by the FRA Director and that they are now much more aware of fundamental rights principles.

This element of the FRA’s work was also showcased through the Maltese Presidency of the Council of the European Union’s invitation of the Director to take part in the informal meeting of the Justice and Home Affairs Ministers in Valletta. This invitation, relatively rare for an EU Agency Director, to participate actively in informal meetings at ministerial level, highlights the importance the Agency has gained over the past decade and the impact it has had and can continue having in policy-making at the

highest level.

4.2.2. Activities and outputs

To what extent are the needs of the relevant stakeholder groups met by the FRA outputs?

To what extent are the Agency’s outputs and activities useful to its various stakeholders?

The purpose of this section is to analyse how relevant FRA’s outputs are to the different stakeholder groups. This includes an assessment of:

 whether the core stakeholders are satisfied with the responsiveness and availability of the research

activities undertaken;

 whether the FRA’s outputs are suitable to the needs of the stakeholders;

 what activities and outputs are seen as most useful by stakeholders; and

 the extent to which different outputs have been taken into account by relevant stakeholders.

Below, each of these aspects will be analysed by referring to the external survey, interviews held with external stakeholders and relevant FRA activities.

Who are FRA’s stakeholders?

Before assessing how relevant FRA’s outputs are for different stakeholders, it needs to be defined who these stakeholders are. According to the Founding Regulation, the key ‘duty bearers’ are the EU-level institutions/agencies and the relevant bodies on the Member State level (i.e. National Liaison officers, national parliamentary focal points and national human rights institutions). Furthermore, FRA stakeholders are international bodies (such as the UN and OSCE), the Council of Europe, and civil society organisations.

However, the Founding Regulation specifically requires FRA to build a dialogue with civil society, 92 which can be interpreted as extending to the rights holders directly. Although rights holders are not the direct target group of the FRA, they are indirectly, through the work of the duty bearers to whom the Agency must provide assistance. Understanding their needs will provide a deeper understanding of the needs the duty bearers and other stakeholders are aiming to address and at the same time provide a better understanding of the context in which the FRA operates. Therefore, the needs of the rights holders are also relevant. It is important to note, however, that the FRA should not be evaluated against the impact it has on rights holders, but solely on the ways in which it addresses the needs of the duty bearers.

92 Article 4(1)(h) Founding Regulation.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

FRA’s Responsiveness

According to the Founding Regulation, FRA should formulate opinions to the Union institutions and to the Member States when implementing Community law, either on its own initiative or at the request of

the European Parliament, the Council or the Commission (also referred to as ‘ad-hoc requests’). 93 FRA

reserves the name ‘opinions’ for self-standing FRA deliverables that comment on legislative drafts proposed at EU level. According to Article 4 para 2 of the Founding Regulation, such opinions are produced on request by the European Parliament, the Council or the Commission. In the period 2009 to 2017 there have been 18 such formal requests for FRA opinions, all of which have been answered by FRA. Six requests were made by the European Parliament in 2016, of which four referred to the EU

94

Common Asylum System. This marks a significant increase in official requests by EU Institutions for legal opinions in 2016, demonstrating FRA’s increasing relevance for current fundamental rights emergencies. Apart from the requests for formal opinions, there were 21 informal requests for FRA input by different European Parliament divisions (MEPs, Legal Service and the LIBE Secretariat) in 2016. Seven of those informal requests related to migration whereas the remaining requests related to a wide range of issues (such as Roma, anti-Semitism, disability and social rights). In regard to most of these requests, FRA could refer back to its previous work and thus answer the request in an efficient

manner. 95 In other cases, no FRA output yet existed. 96 While MEPs have not directly commented on the

value of those informal requests, one interviewee from the EP has mentioned that a very important EP report on democracy, rule of law and fundamental rights relied heavily on FRA information – showing its

relevance.

According to Articles 8–10 of the Founding Regulation, the Agency shall ensure close cooperation with Member States, civil society and international organisations. While FRA is not required to reply to formal requests for opinions by those stakeholders, it is evident that FRA has been increasingly responsive to their needs. For instance, in 2016 FRA brought together its network of national stakeholders and FRA bodies in Vienna to identify more opportunities for cooperation and greater synergy. Furthermore, FRA has increased its cooperation with national parliaments. 97 The Director has also undertaken numerous

country visits to continue strengthening collaboration with, among others, civil society and the media. 98 Furthermore, FRA has since 2013 assisted four Member States in implementing certain policy areas relevant to fundamental rights. 99 Additionally, on the international level many activities have been pursued over the years to ensure that FRA is responsive to their needs. 100 More information on this can be found in section 4.4, ‘Assessing coordination and coherence’.

Matching needs of stakeholders

Around 57% of the respondents to the external survey stated that the interaction with the FRA and its outputs meets their needs to a large extent. A considerable minority of respondents in both surveys

argued that their needs are only met to a moderate extent (i.e. 33% respondents). Only a small minority

93 Article 4(1)(d) Founding Regulation.

94 Annual Activity Report 2016.

95 For example, a request on aging was dealt with by referring to a FRA publication (‘Addressing barriers preventing older people from living

independently in the community’).

96 For example, a request on social rights in Europe could not be based on evidence collected by FRA.

97 Annual Activity Report 2016, p. 11

98 Annual Activity Report 2016, p. 12

99 Bulgaria, Czech Republic, Slovakia, Spain and Germany.

100 See more details under EQ 19

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

in both external surveys felt that FRA only meets their needs to a small extent, or not at all. 101

Furthermore, most survey respondents mentioned that FRA was flexible enough to adapt to changing needs between 2013 and 2017. In the external surveys, around 40% found that FRA was to a large extent flexible enough. Furthermore, in the stakeholder interviews, interviewees from the Council, Commission and European Parliament also confirmed that FRA was flexible to adapt to current

fundamental rights challenges.

Respondents to the external survey 102 further emphasised the following areas of fundamental rights needs that were particularly important in the period 2013–2017: Equality and non-discrimination; Racism, xenophobia and related intolerance; and Immigration and Asylum (see Figure 12). In terms of outputs, most stakeholders that were interviewed identified their needs as relating to FRA data and

analysis on fundamental rights issues, as well as opinions and recommendations.

It was mentioned multiple times that the new FRA Director in particular is successful in interpreting the FRA mandate broadly and thus responding to the changing

‘ We should commend the Director and needs of civil society. One interviewee mentioned that ‘under

the Management Board for interpreting

the mandate broadly in order to achieve the previous and current director, [FRA has] made great strides

some goals that were unexpected at the to become a more relevant organization’ in addressing

time the Agency was launched. ’ fundamental rights concerns. One stakeholder of the UNHCR pointed out that FRA has become a much more relevant

Former Commission Official institution in relation to migration. This is exemplified by the

FRA’s regular monthly overviews of migration-related fundamental rights concerns, which is a meaningful output for the UNHCR EU office and global offices. The latter example illustrates that FRA objectives have been flexible enough to adapt to changing fundamental rights needs and thus become a meaningful institution for international stakeholders.

FRA’s relevance on migration issues: Regular overviews of migration-related fundamental rights concerns

Since the EU has in recent years experienced the arrival of a large number of refugees, asylum seekers and migrants, the European Commission asked FRA to collect data about the fundamental rights situation of people arriving in those Member States that have been particularly affected by migration. The countries scrutinised by FRA include: Austria, Bulgaria, Denmark, Finland, France, Germany, Greece, Hungary, Italy, the Netherlands, Poland, Slovakia, Spain and Sweden. Starting in January 2016, FRA has been publishing monthly updates on among others: reception conditions upon arrival, registration procedures, criminal proceedings, and racist incidents. The overviews have been well received by a number of different stakeholders and show FRA’s flexibility to adapt to the current needs of its duty bearers and rights holders. For example, a stakeholder from the UNHCR mentioned that the UNHCR regularly reviewed the overviews. Furthermore, a representative of a national human rights institution and an NLO argued that the regular overviews are showing how responsive FRA is in reacting to current challenges.

Apart from this overall positive view on the relevance of FRA’s outputs, some stakeholders mentioned that some structural changes could render FRA even more relevant to its stakeholders. For instance, it

101 In both cases under 8% on both aspects.

102 Respondents to the national survey were in most cases staff of the national human rights bodies or NGOs.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

was mentioned that since the decision on FRA’s research focus is made too long in advance means that

outputs are less relevant when they are finalised. 103 Furthermore, some civil society and national-level representatives mentioned that FRA should focus on fewer research areas. 104 Many stakeholders such as

NLOs, civil society organisations, the Council of Europe and the EP were critical of the fact that FRA has no competences to initiate research in relation to criminal and police cooperation even though it is the field which is most fundamental-rights sensitive. 105 The usefulness of a potential role for FRA in criminal and police cooperation is underlined by the fact that both the EP 106 and civil society 107 have advocated

for an extension of FRA’s remit in this regard.

While most interviewees commented positively that FRA’s output and research is highly independent and unbiased, not as many mentioned FRA’s independence in regard to agenda setting. Half of those few interviewees that commented on this aspect also felt that FRA as an organisation is sufficiently independent. However, the other half had mentioned that more independence of FRA would be useful. For example, one Council member and an NLO representative argued that FRA should be more independent from the political level during the multi-annual agenda-setting through the process of adopting the MAF. One civil society representative even mentioned that FRA should have extended powers in order to be more relevant, such as being able to sanction Member States for fundamental rights violations or to give binding opinions to the Council and the Commission. Given the nature and

mandate of EU agencies this is, however, not realistic.

Usefulness of FRA outputs (i.e., have stakeholders taken FRA’s outputs into consideration?)

At EU level, FRA outputs have proven to be very useful to its stakeholders. For example, in December 2016 the European Parliament adopted a resolution on the situation of fundamental rights in the European Union in 2015, which contains 20 references to the findings and work of FRA. More specifically, the EP quoted FRA statistics and findings on violence against women, Roma integration, and LGBTI rights. 108 Apart from the use of data, the report also encourages FRA to continue its work in several areas and stressed that more use shall be made of FRA’s expertise and outputs. 109 In contrast, the narrative of the 2009 EP Fundamental Rights Report was based more on encouraging the use of

110

FRA’s outputs. This indicates that the usefulness of FRA’s outputs has increased in recent years.

103 National Liaison Officer

104 Only one NGO representative mentioned that FRA should completely focus on anti-discrimination since this was its purpose when it was set

up. One NLO mentioned that FRA should emphasise the research on areas where there are bottlenecks for EU decision-making and

that, for example, its research on privacy is not needed.

105 Note that FRA is not allowed to conduct research on former third pillar measures on its own initiative. However, FRA is able to conduct

research if requested explicitly by the EU Institutions or bodies.

106 For example, Working Document on Establishment of an EU mechanism on democracy, the rule of law and fundamental rights – Scoreboard

on Democracy, Rule of Law and Fundamental rights Committee on Civil Liberties, Justice and Home Affairs, 9.2.2016, Brussels.

107 An open letter of several fundamental rights bodies requesting the extension of FRA’s mandate.

http://www.europapraw.org/files/2011/10/Open-letter-on-the-need-to-extend-the-competences-of-FRA.pdf

108 Report on the Situation of fundamental rights in the European Union in 2015 (2016/2009). For example, paras.: Z, AJ, AO, AP, 123

109 Report on the Situation of fundamental rights in the European Union in 2015 (2016/2009). For example, Paras.: 53, 94, 101, 122;

110 European Parliament resolution of 15 December 2010 on the situation of fundamental rights in the European Union (2009) – effective

implementation after the entry into force of the Treaty of Lisbon (2009/2161(INI) i), para. 33 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Furthermore, in 2016 both the Council Conclusions on the application of the Charter 111 and the European Commission Report on the Application of the EU Charter of Fundamental Rights 112 made

references to various FRA outputs; the Council Conclusions particularly encourages Member States to

make more use of the FRA in training activities. 113 FRA’s outputs were also reflected in EU legislation.

The European Commission’s proposal for a Directive on European Criminal Records Information System (ECRIS) to allow information on criminal sentences of third-country nationals to be exchanged took into account safeguards that FRA suggested in its opinion of December 2015. 114 However, some interviewees have expressed doubts in concluding that FRA outputs directly influence policy outcomes since policy

discussions are not transparent and can be influenced by various different factors. 115

In various stakeholder interviews, there was also discussion on how far FRA outputs are useful for national authorities and national civil society organisations, and international organisations and civil society organisations. It was mentioned a few times that FRA outputs are used by national governments for informational purposes rather than to inform the policy-making process. For example, one interviewee mentioned that FRA publications are regularly circulated in the relevant departments of the national ministry and all officials are advised to read those FRA reports. The interviewee further mentioned that the extent to which FRA outputs are then used at national level depends heavily on the national agendas. For example, if governmental authorities prepare a bill which can be supported by FRA data, the respective FRA outputs are likely to be used and referenced. On the contrary, if FRA outputs do not support the bill they are not likely to be used. On the international level, FRA outputs in

particular have been used on Roma integration by UN and CoE bodies. 116 In respect to civil society

organisations, it has often been mentioned that NGOs are very interested in FRA outputs for their own purposes due to their limited capacity to collect data. 117 Furthermore, FRA’s ability to aggregate data

from different regions in one country is considered useful.

To conclude, it has been shown that FRA is to a large extent relevant to the needs of key stakeholders. First, FRA has been responsive to stakeholders by providing advice, and engaging with stakeholders. Second, the FRA’s outputs coincide with the key needs/interests of most stakeholder groups both in terms of thematic areas (especially migration) and the type of output (e.g. surveys, evidence). Finally, there has been some evidence that FRA outputs have been directly included in outputs of its

stakeholders, illustrating its usefulness.

FRA’s work on Roma

In the period 2013 to 2017, FRA invested heavily in Roma integration. As outlined in the strategic plan, FRA invested in Roma integration by pursuing five strategic objectives including the development of robust data collection methods; the support of the European Commission and Member States; the

111 Council Document 8946/16

112 SWD (2016) 158 final

113 Council Document 8946/16, para. 8.

114 Proposal for a DIRECTIVE OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL amending Council Framework Decision 2009/315/JHA, as

regards the exchange of information on third country nationals and as regards the European Criminal Records Information System

(ECRIS), and replacing Council Decision 2009/316 i/JHA (COM/2016/07 final)

115 Advisory Panel interviewee

116 Annual Activity Report 2016, p. 19.

117 Interview with civil rights organisation.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

identification/promotion of good practices of local Roma integration; establishment of dialogue; and the encouragement of Roma participation in designing and implementing integration strategies. Those strategic objectives were all re-confirmed in the Mid-Term Review of 2015.

FRA was successful in regard to all of its five objectives. For example, FRA has developed robust data collection methods and has collected data in the field of Roma integration. The value of these activities and FRA’s other Roma-related outputs is reflected by the many instances in which FRA has been referenced by EU stakeholders. For example, after the results of the EU-MIDIS II: selected results (Roma) were published at the European Roma Platform on 29 November 2016 it was also presented to the LIBE Committee hearing. 118 Furthermore, the results of the survey also fed into the country progress reporting of five countries in the context of the European Semester. 119 Notably, this was the only FRA data used for the European Semester, illustrating the importance of FRA work in this field. In addition, the 2016 report from the European Commission entitled ‘The state of European cities 2016: cities leading the way to a better future’ includes some of FRA’s findings on Roma. Furthermore, upon request, FRA contributed to the drafting of Council Conclusions on accelerating the process of Roma integration and to Council Conclusions on the European Court of Auditors’ special report on EU policy initiatives and financial support for Roma integration. Last but not least, the Commission Communication on mid-term evaluation of Council Recommendation of 2016 on Roma makes use of FRA data to report on Member State performance. 120

FRA’s activities also have a direct impact on Member States. Following requests, FRA has assisted Bulgaria, the Czech Republic and Slovakia and Germany in developing its monitoring and evaluation system in respect to Roma integration. It should be noted that in the period from 2013 to 2017 FRA was only requested by five Member States to provide assistance in the area of fundamental rights. Four out of these five requests relate to Roma integration. All requests were met either through missions or support provided online. One particular area of support was the preparation for the Slovak Presidency of the Council. FRA assisted the Slovak Presidency preparatory team in defining the topics to be covered during the Presidency in the area of Roma inclusion. 121 All of these requested support services illustrate how relevant FRA’s work on Roma is for Member State authorities.

FRA has also had a considerable impact on the local level. For example, in 2013 a three-year-long fieldwork project – the Local Engagement for Roma Inclusion (LERI) project – was initiated. During the

lifespan of the project tailored interventions took place in all participating localities and evidence of Roma integration at the local level was collected. At the end of 2016 final case studies for 22 localities participating in the project were presented including locations in Bulgaria, the Czech Republic, France, Greece, Hungary, Italy, Romania, Spain and the UK. The case studies outline the identified local needs, the local project plan design and implementation, and the various local-level interventions and their outcomes. 122 By comparing the needs and projects in various Member States, FRA established a dialogue with local communities and identified good practices on the local level.

118 Annual Activity Report 2016, p. 19.

119 Annual Activity Report 2016, p. 19.

120 Communication from the commission to the European Parliament, the Council, the European Economic and Social Committee and the

Committee of the Regions Assessing the implementation of the EU Framework for National Roma Integration Strategies and the

Council Recommendation on effective Roma integration measures in the Member States — 2016 COM (2016) 424 i, 27 June 2016

121 Annual Activity Report 2015, p. 25.

122 Annual Activity Report 2016, p. 19.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

The examples quoted above illustrate in a non-exhaustive manner FRA’s activities and outputs in the field of Roma integration and the impact of those activities and outputs. It becomes clear that FRA had a considerable impact on EU level, as reflected by various stakeholder and policy documents referring to FRA’s outputs on Roma integration. Furthermore, it has also been shown how FRA’s work has had a positive impact on the local and national levels.

4.2.3. To what extent have the objectives set out in the multi-annual and annual work programmes for the years 2013 to 2017 been accomplished?

The aim of this section is to analyse the extent to which the strategic and thematic objectives set out in the Agency’s Strategic Plan and the annual work programmes for the years 2013 to 2017 have been accomplished. Furthermore, the aim is to assess in how far the activities of the Agency were adapted in order to better respond to the multi-annual strategic and thematic objectives.

In 2015 progress made towards the achievements of the thematic (and strategic) priorities in the Strategic Plan was assessed in the Mid-Term Review of the FRA Strategic Plan 2013–2017. Although the

Mid-Term Review ‘largely reaffirmed the existing strategy’, it set the revised objectives and expected results, as well as what the main operational focus of the FRA should be in 2016–2017 for each thematic area. The purpose of the slightly revised objectives is to completely fulfil the set objectives by the end of 2017 and ‘to respond to the context changes and emerging needs for assistance and expertise in FRA’s areas of work’. For example, responding to the arrival of asylum seekers in large numbers in the EU has led to a more intense focus on outputs in the area of migration and asylum. Closely related to this point is that the response to current emergencies shall however be well balanced with the need for long-term research planning. Apart from this, some aspects shall be strengthened on cross-cutting issues. For instance, communication with stakeholders should be given more priority through awareness-raising and FRA shall ensure the dissemination of its work to empower multipliers (e.g. contact points and national stakeholders).

Thematic objectives

In all thematic areas, the key targets can broadly be summarised as providing advice to relevant stakeholders, collecting data and developing methodologies to analyse data, establish relationships with key stakeholders and raise awareness among the people affected by human rights violations. FRA largely achieved those targets in regard to all thematic objectives as originally set in the Strategic Plan and as revised by the Mid-Term Review. For example, FRA regularly published reports on all thematic

areas and provided updates on all fields in the Annual Fundamental Rights Report, 123 it supported the EU Institutions with formal opinions and informal advice, 124 it strengthened cooperation with key

125

stakeholders, and it informed stakeholders and affected persons about the respective rights. 126 This

123 Fundamental Rights Reports 2013–2017 and all thematic specific reports.

124 For example: FRA Opinion on the situation of equality in the European Commission 10 years after the equality directives have been adopted.

Another example is that the Council adopted a directive on freezing and confiscation of proceeds of crime in March 2014 that has

taken into consideration several aspects of the FRA opinion on fundamental rights.

125 For example, FRA set up a Working Party on Improving Recording and Encouraging Reporting of Hate Crime, which brings together FRA, all 28

EU Member States, the European Commission, the Office for Democratic Institutions and Human Rights (ODIHR) and the European

Commission against Racism and Intolerance (ECRI) at the Council of Europe.

126 For example, FRA developed 28 human rights indicators to assess the political participation of persons with disabilities in the EU.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

has also been confirmed by respondents to the external stakeholder survey (see Figure 12). In all but

one case stakeholders mentioned that that the Agency met its thematic objectives to a large extent. 127

The key challenges experienced across all topic areas related to the lack of prioritisation of the topics at political level and thus missing opportunities to provide advice, and the lack of resources to get stakeholders in all 28 Member States involved. Furthermore, from a methodological point of view, it was often difficult to ascribe an achievement to only one thematic objective due to its cross-cutting nature. 128 A more detailed overview of the fields of migration, the rights of the child and Roma integration is given below, due to their current relevance.

Figure 12: External Survey: In your opinion, to what extent has the Agency met its thematic objectives? (N=94)

Discrimination 49 28 2 0 4

Immigration 46 22 4 1 9

Racism 43 28 3 1 7

Right of the child; 40 30 3 0 7

Roma Integration; 38 29 5 02

Victims of crime, including compensation to victims; 37 29 1 1 17

Access to justice; 32 29 3 1 17

Information Society; 28 28 3 2 19

0 10 20 30 40 50 60 70 80 90

To a large extent To a moderate extent To a small extent Not at all Do not know

In respect to migration and asylum, the core objectives laid down in the Strategic Plan 2013 to 2017 relate to FRA providing relevant and timely advice, producing evidence, sharing best practices, advising and supporting Frontex and EASO, coordinating with UNHCR and to contributing to the development of policies. In the Mid-Term Review, it was mentioned that due to the on-going crisis situation in the area of migration it is necessary to prioritise cross-cutting issues between thematic areas, and those areas where there are serious violations of human rights. Furthermore, FRA shall consider partnerships with other relevant institutions apart from Frontex, reinforce collection/dissemination of best practice, and take into account the capacity problems of FRA in the field given the on-going crisis. Reviewing FRA’s activities over the last five years it is evident that FRA largely achieved those objectives.

First, FRA carried out relevant research, particularly on the operational level. 129 Furthermore, FRA

submitted four legal opinions to the European Parliament and provided informal advice to the European

Parliament on numerous relevant aspects in relation to asylum. 130 Additionally, FRA published monthly

overviews of migration-related fundamental rights concerns with respect to the situation on the ground in 14 Member States. The advisory role and research activities show that FRA managed to develop into

127 Only in regard to Information Society, the same amount of stakeholders felt that FRA met its objective to a large extent and to a moderate

extent (28).

128 Stock-taking reports.

129 E.g. FRA 2015 Report on labour exploitation of migrant workers, three publications on external borders in 2013 and 2014; 2015 report on

Fundamental Rights implications of large-scale processing of biometric data of migrants.

130 Annual Activity Report 2016, p. 8.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

‘a key actor providing robust, relevant and timely advice’. 131 FRA also demonstrated sufficient flexibility

to account for the objectives in the Mid-Term Review in 2015. For example, FRA managed to establish

relationships with other relevant organisations such as eu-LISA, Europol and CEPOL. 132 Furthermore, FRA

contributed to the draft EASO tool on ‘Best interest assessment for the purpose of relocation of unaccompanied children’ accounting for the Mid-Term Review’s suggestion on combining migrationrelated

rights and other rights. 133

Apart from the overall positive situation, the main challenge faced by FRA in relation to migration and asylum related to capacity problems. While various activities have been carried out, others had to be

declined due to a shortage of resources. 134

In respect to the rights of the child, the Strategic Plan 2013 to 2017 sets nine specific objectives on the rights of children including, among others, the collection of primary and secondary data, assisting the EU Institutions, and developing best practices. The Mid-Term Review suggested that the focus be shifted to: mainstreaming the rights of the child across all the thematic areas; research on how Member States implement EU law relevant to the rights of the child; and to solid cooperation with the EP and the Commission. FRA achieved the objectives of both the Strategic Plan and the Mid-Term Review. Most

notably, FRA has conducted a mapping of child protection systems in EU28, 135 provided expertise to institutional stakeholders to assist them in fulfilling child rights across different areas, 136 and successfully

137

conducted cross-topic research by prioritising children in migration.

The key challenges in regard to FRA’s work on the rights of the child relate to coordinate its work with EU Institutions due to dispersed competences on the rights of the child. It was also pointed out that the

cooperation with OHCHR and national bodies was limited.

In respect to Roma integration, the Strategic Plan 2013 to 2017 sets out five strategic objectives including the development of robust data collection methods that allow comparative analysis of the situation of Roma across the EU; the support of the European Commission and Member States to establish and improve monitoring mechanisms for Roma integration strategies; the identification/promotion of good practices of local Roma integration; establishment of dialogue on Roma integration; and the encouragement of Roma participation in designing and implementing integration strategies. The Mid-Term Review of 2015 reiterated all five objectives until the end of the

2017.

FRA largely achieved its strategic objectives in the period 2013 to 2017. For example, in 2015 FRA started to implement its qualitative research on the local level, collecting data through case studies in 22

131 Strategic Plan 2013–2017, p. 40.

132 Annual Activity Report 2016, p. 8.

133 Ibid.

134 Stock-taking report migration

135 Annual Activity Report 2015 and 2016

136 FRA also gave inputs to the Council Working Party on Substantive Criminal Law (DROIPEN) on procedural safeguards for children suspected or

accused in criminal proceedings as well as to the FREMP on Rights of the Child. The Council of Europe (CoE) European Committee on

Legal Cooperation invited FRA to become a member of the informal network on child-friendly justice to develop a strategy to assist

national actors in implementing the CoE guidelines on child-friendly justice

137 For instance, the FRA monthly reports on the migration situation in March, June, September and December specifically focused on children

and issues that affect children strongly, such as gender-based violence and family reunification (Annual Activity Report 2016, p. 21).

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

municipalities on best practices and ways to improve current practices. 138 Furthermore, the report EU-

MIDIS II: selected results (Roma) was launched at the European Roma Platform on 29 November 2016

and presented to many different stakeholders including the LIBE Committee and the Commission. 139

Furthermore in 2016, by request, FRA contributed to the drafting of Council Conclusions on accelerating the process of Roma integration and to Council Conclusions on the European Court of Auditors’ special report on EU policy initiatives and financial support for Roma integration. In 2016 FRA also supported the European Commission by providing two capacity-building workshops to DG REGIO which led to

140

follow-up requests for training.

While largely achieving its strategic objectives, the Agency has experienced some challenges, including the need to continuously encourage certain stakeholders to make use of FRA outputs in relation to Roma integration. Another challenge was to develop approaches which are widely shared between

Member States in view of the different situation in each country. 141

Strategic Objectives

142

In addition to the thematic objectives, the Strategic Plan for 2013–2017 lays down the Agency’s Strategic Objectives. They cover a range of activities and goals that the FRA is aiming to achieve over the defined period. The following table summarises the Strategic Priorities and Focus Areas mentioned in

the Mid-Term Review.

Strategic Priorities 2013–17 Mid-Term Review 2015 Focus Areas

Strategic Priority 1: Enhancing  Develop synergies and avoid duplication with JHA EU Agencies FRA’s contribution the  EU Institutions shall use FRA’s research more extensively in policyprocesses at EU level making process  Close cooperation with Commission shall continue Strategic Priority 2: Enhancing  Engage with MSs (raise awareness, share good practices, provide

FRA’s contributions to processes assistance)

at national level  Explore further engagement with civil society

 Strengthen the FRA’s response to MS’s requests for assistance and

expertise

 Update the Agency’s legal handbooks for court practitioners  Support national authorities in developing methodologies for collecting

data

Strategic Priority 3: Identifying  Analyse relevant data trends over time and measuring  Explore ways of assessing trends

progress in Member States  Further develop human rights based indicators

 Continue to enhance/develop interactive maps

Strategic Priority 4: Developing  Respond to emergencies as appropriate

timely and targeted responses  Balance long-term planning with need of emergencies

to fundamental rights emergencies

Strategic Priority 5: Improving  Raise awareness among different stakeholders (police, media, etc.) the impact of FRA  Ensure availability of FRA outputs at national level communication and awareness 

raising Disseminate FRA outputs through events

138 Stock-taking report on Roma Integration 2015

139 Annual Activity Report 2016

140 Annual Activity Report 2016

141 Stock-taking report on Roma Integration 2015.

142 FRA Strategic Plan 2013–2017, http://fra.europa.eu/sites/default/files/fra_strategic_plan_en.pdf

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Strategic Priorities 2013–17 Mid-Term Review 2015 Focus Areas

 Diversify awareness-raising products

Strategic Priority 6: Planning  Develop new indicators to assess FRA’s performance FRA work and evaluating its  Establish best-practice indicators impact  Repeat the stakeholder review and external evaluation

The Mid-Term Review mentions what the main operational focus of the FRA should be in 2016–2017 for each strategic priority, in order to fulfil all the objectives set by the end of 2017 and ‘to respond to the context changes and emerging needs for assistance and expertise in FRA’s areas of work’. The priority areas identified in the Mid-Term Review have across all priorities largely been implemented in the subsequent period. For example, in accordance with Strategic Priority 3 the most obvious way FRA reacted to emergencies relates to its activities on migration. On top of its planned activities, FRA published monthly reports and focused on the most pressing issues such as unaccompanied minors,

migrant smuggling, etc. 143 Another example relates to Strategic Priority 5 where FRA achieved its goals by, among other things, organising its biggest event with 700 participants. 144

In regard to all but two areas, the external survey respondents believed that FRA met its strategic objectives to a large extent. Only in relation to communication and awareness, as well as FRA’s contribution to national-level processes, did stakeholders feel that FRA achieved this objective only to a moderate extent.

Figure 13: External Survey: In your opinion, to what extent has the Agency met its strategic objectives? (N=94)

Contribute to national processes 13 33 19 4 13

Identify tends 28 35 11 0 8

Communication and Awareness Raising 33 34 9 0 8

Evaluation 34 27 4 0 17

Respond to emergences 40 23 10 1 8

Contribute to EU level processes 48 20 3 1 12

0 10 20 30 40 50 60 70 80 90

To a large extent To a moderate extent To a small extent Not at all Do not know

An important and challenging priority is FRA’s contribution to the national level (Strategic Priority 2). Overall, FRA increased its cooperation with national stakeholders. For instance, FRA has presented its annual findings to several national parliaments and in 2015 FRA established national contact points in each national Parliament. 145 Furthermore, FRA has intensified its cooperation with NLO, leading to

143 Annual Activity Report 2016, p. 13.

144 FRA Fundamental Rights Forum 2016.

145 Annual Activity Report 2015.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

further awareness-raising events about FRA outputs being organised by NLOs. In addition, since 2015 FRA has initiated thematic and national workshops to increase awareness of FRA output in the framework of FRP One indicator showing the positive results of the intensified cooperation is the

number of FRA publications downloaded. In 2014, FRA was the EU agency with the largest number of downloads in different languages, showing its relevance to different stakeholders. 146 An example of where FRA not only added value on the national level but where national practices positively influenced

FRA relates to the rights of the child. In one interview, it was mentioned that after attending a conference at national level where children were present, FRA adopted the practice of inviting children

to discussions on children’s rights. 147

Apart from these positive aspects, it will be important that FRA further increases the uptake of its work by national-level stakeholders. In order to increase FRA’s

‘ The Agency uses its mandate in a relevance on the national level, the Mid-Term Review suggests flexible way in order to be able to that FRA should engage more with intermediaries (such as respond to changing needs of the EU CSOs and NGOs) who are often well connected with national Institutions. ’ institutions; FRA should develop a tailor-made approach for

engagement activities with each Member State 148 to account

Civil Society representative for political sensitivities on certain topics; and it should step up

with capacity and training activities. 149 While FRA has on multiple occasions reported its research findings on the national level 150 , from 2013 to 2015, FRA was

asked to provide expertise to five Member States (Bulgaria, the Czech Republic, Slovakia, Spain and Germany) on Roma integration, hate crime and children in criminal proceedings. 151 At the same time,

additional expertise related to specific projects greatly increased the number of interactions the Agency has at national level. With respect to the Violence Against Women Survey, for instance, over 20 country

visits were undertaken to report on the findings at Member State level, including an invitation to provide expertise to the German Bundestag. In the case of the FRA’s work in migrant hotspots, the Memorandum of Understanding signed with Greece means the Agency systematically aids the

government, so that the work of the FRA is having a direct and immediate impact in a crisis. While it is not necessarily within the mandate of FRA to provide assistance extensively to Member States, an increase in requests for assistance may be a good indicator to assess FRA’s visibility on the national

level. Apart from that, it was also mentioned that FRA’s visibility on the academic level should increase. While several academic institutions have already included FRA outputs in their libraries, there is scope

152

for improvement.

To conclude, FRA was largely successful in achieving both its thematic and strategic objectives as set in the Strategic Plan from 2013 to 2017. In addition, FRA has proven to remain sufficiently flexible to

146 Stock-taking report on Strategic Priority 2.

147 Is Europe doing enough to protect fundamental rights? – the children’s perspective – 28 June 2017.

148 Note that FRA seems to adopt a similar approach as other EU agencies such as the EMCDDA. Both agencies rely on conducting visits in EU

Member States including institutional visits, attending relevant events and participating in parliamentary hearings (see EMCDDA

General Report of Activities 2016). It needs to be noted, however, that the topics EMCDDA is dealing with may be less politically

sensitive than topics addressed by FRA.

149 Mid-Term Review, p. 8.

150 For example, the Violence against Women survey results were presented to several national parliaments, where FRA was invited to give

evidence.

151 No information is available on whether there have been more requests from 2015 to 2017.

152 Annual Report 2016, p. 13.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

account for the slightly adapted priorities set by the annual work programmes and the Mid-Term Review in 2015. While this section has highlighted some challenges both in relation to the thematic and

strategic objectives, it has to be noted that these challenges mainly related to aspects where FRA’s performance depended on external factors such cooperation with international organisation and its

relevance to inform EU policy-makers.

The Agency uses its mandate in a flexible way to be able to respond to changing needs of the EU Institutions and the Agency is quite good at making sure they pick the urgent needs at EU level to work on.

4.2.4. To what extent are FRA’s outputs fully accessible and made use of by relevant stakeholders?

This evaluation question involves two aspects: First, the extent to which FRA disseminates its research findings to raise awareness among its key stakeholders needs to be assessed. This involves an

assessment of the Agency’s dissemination activities on social media (including Twitter, Facebook, YouTube, FRA website) and via meetings and events but also an assessment on how often references

are made to FRA in research, policy and journalistic publications. 153 Second, it needs to be assessed to what extent FRA outputs are accessible by end users involving an assessment on how easily information

is found online and how many language versions exist. The third aspect of the question ‘how much do relevant stakeholders make use of FRA outputs’ is largely covered under EQ3. While section 2.2.4

analyses accessibility and dissemination of FRA outputs generally, in the appendices of this report the accessibility and dissemination of FRA outputs is explained in further depth by focusing on the social

media campaign to release the EU-MIDIS II report.

Accessibility of FRA outputs

To start with it is crucial to assess how accessible FRA outputs are in a practical sense. All FRA outputs are publicly available from its website and they can also be

‘ FRA objectives have been flexible ordered from the EU bookshop. Some of FRA’s outputs are enough to adapt to changing published in all or several EU languages while most of FRA’s

fundamental rights needs, publications are published in English only. In the stakeholder particularly with the migrant crisis ’ interviews, some interviewees have argued that the fact that

UNHCR representative most FRA outputs are only available in English does not have

an impact on the accessibility of FRA outputs since FRA’s target audience is mostly using English as their work language or are at least fluent in English. Thus, instead of investing unnecessarily in translation, FRA should rather invest in its research activities. However, other stakeholders have pointed out that in countries like Italy it is not necessarily the case that everyone will speak English. Translating more outputs into all EU languages was thus considered beneficial.

It is also interesting to assess the trends in relation to what has been downloaded from FRA’s website as shown in Figure 14. In some thematic areas, downloads are consistently high over the years. For

example, from 2014 to 2017, the Violence Against Women Survey has been the top download with over 20,000 downloads each year (and 150,293 downloads in the period 2014 to July 2017). Furthermore, in

153 Note that this section focuses on research publications. References to FRA in policy documents (section 4.2.2) and media (spread in various

parts) are addressed in other sections of this report.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

the same period the LGBTI survey results have also been popular. While in the graph below the LGBTI thematic area is only fourth, in 2013 it was downloaded most often. Furthermore, in other fields some peaks can be detected. For example, in 2016 a high number of downloads related to the thematic priority of migration and asylum. The asylum handbook has been downloaded more than 12,000 times and the handbook on anti-discrimination has been downloaded more than 27,000 times. Over the total period migration and asylum outputs have been downloaded 47,525 times. This can be linked to the

current needs of many of FRA’s stakeholders in the area of migration and asylum. One representative of

the UNHCR pointed out that FRA’s monthly Regular overviews of migration-related fundamental rights concerns is a meaningful output for the UNHCR EU office and global offices. In sum, it can be concluded

that FRA is an established source for information in some fields but is also widely consulted on topics of current relevance.

Figure 14: Downloads in the period 2013–July 2017 per thematic area 154

160,000 140,000 120,000 100,000

80,000 150,293

60,000

40,000 71,593 65,439 20,000 35,768 27,222 9,479 4,313 3,626 3,467

0

Dissemination of FRA outputs

Apart from the question of accessibility in the more technical sense, it is also crucial to understand how far relevant stakeholders are made aware of the usefulness of FRA outputs. FRA has over the last five

years been very successful in disseminating its activities and in raising awareness of its outputs among its key stakeholders.

First, FRA has been active on social media such as Facebook, Twitter, YouTube and (to a lesser extent) LinkedIn. 155 In all of these different platforms a steady increase of activities, followers/fan and likes can be noted. For example, on Facebook the FRA page received only 702 likes in 2012 but this number increased steadily over the years (i.e. in 2013 there were already 3,939 likes and in 2016 the number

154 It needs to be noted, however, that in some thematic areas more outputs have been produced than in others. Therefore, the graph does not

necessarily reflect that some fields are more popular than others. The category ‘Fundamental Rights’ refers to the downloads of the fundamental rights reports.

155 Note that no access is available to data on LinkedIN dissemination activities.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

increased further to 7,247). In addition, the number of fans increased constantly over the years. While in 2013, 113,971 Facebook users were fans of FRA’s Facebook page, in 2016 this number increased to 126,196. While slightly lower in numbers than Facebook, an increase of Twitter followers can also be detected from 2013 to 2016. While in 2013 64,307 persons followed FRA on Twitter this number increased to 70,439 in 2016. While views, likes and average watch time of FRA’s YouTube videos have not been increasing as steadily, 156 it is still noteworthy that since 2013 FRA videos have been viewed over 80,000 times. All in all, it can be concluded that FRA has been successful in disseminating its activities over social media and a positive trend can be detected over the years. This is well illustrated in

Figure 15, which shows the steep increase of followers on different social media platforms.

Figure 15: Social media followers on different social media platforms over the years 2010–2016

50000 Facebook page

45000 likes

40000 Twitter

35000 followers

30000 Instagram

25000

20000 LinkedIn

15000 10000 5000 0 2010 2011 2012 2013 2014 2015 2016

Second, FRA also engages in more traditional dissemination activities such as organising and participating in events. In accordance with the objectives laid down in the Strategic Plan 2013–2017, FRA has hosted and participated in a variety of events in each of its thematic priority areas and has held

meetings with multiple stakeholders on a bilateral basis. 157 Probably the most impactful event in this

respect is the Fundamental Rights Forum, which was held for the first time in 2016 and will take place biennially. The 2016 Fundamental Rights Forum was three days long, had the title ‘Rights, respect, reality: the Europe of values in today’s world’ and was attended by 700 participants. The aim is to foster discussion among leading experts, policy-makers and practitioners on all fundamental rights topics. The

forum gave rise to more than 100 practical ideas and policy proposals. 158 A high-profile event like the

Fundamental Rights Forum is an excellent way to disseminate FRA’s activities and make them accessible to stakeholders. The effect the Fundamental Rights Forum has on the dissemination activities is well illustrated with a peak of visits on the FRA website during the period of the Fundamental Rights Forum in June 2016 (see Figure 16). While there may of course be other reasons that may have triggered the

peak of visits, the timing suggests the positive effect of the Fundamental Rights Forum.

156 There are rather several increases/decreases throughout the years.

157 For examples in relation to the different thematic priority areas, see the annual activity reports 2013 to 2016.

158 Annual Activity Report 2016.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

The Fundamental Rights Forum, June 2016

The Fundamental Rights Forum was implemented in the FRA’s thematic area of ‘cross-cutting projects and activities’ 159 during the 2016 period as 1st priority project. The aim was to raise public awareness of fundamental rights and to create a fruitful dialogue and close cooperation with relevant fundamental rights stakeholders. In this way, it implements Article 4 and 19 of the Founding Regulation. The Forum

can be considered as a success both in terms of its implementation and the emerging follow-up activities.

In regard to the implementation, the forum was attended by 700 participants with a diverse background creating an environment where experts, policy-makers and practitioners from various fundamental rights fields could exchange views. In this way, the FRF filled a clear gap since no other event so far was able to bring such a diverse group of stakeholders together (whether on national or EU level). The added value of the FRF has been confirmed by the positive reactions on social media and the active

participation of all attendees in the sessions of the FRF. Furthermore, participants defined the FRF as:

a significant moment in a modern human rights movement, a milestone that has potential to boost

the implementation of a fundamental rights culture in the EU 160

Apart from the successful implementation the follow-up phase also indicates the positive effects of the FRF. The key conclusions of the events were summarised in the Chair’s statement. The Statement lists proposals for action categorised according to the three core topics of the Forum (i.e. migration, information society and discrimination). All of the proposals for action fit neatly into FRA’s mandate and can be accommodated in FRA’s work. For example, in relation to information society, the FRF concluded that both the negative and positive aspects of the digital age need to be addressed by FRA. The project ‘Additional activities in the area of Information society and, in particular, respect for private life and protection of personal data’ in the Annual Work Programme 2017 and 2018 examines further the role of ICT with respect to its impact on key fundamental rights – looking at both the negative and (potentially) positive implications of increased use of

ICT.

The success of the event on different levels can be ascribed to the considerable planning efforts that preceded the event. The project went through all key steps for an informed and structured planning (Ex-ante evaluation, FRA Project Planning Evaluation (FRAPPE), stakeholder consultation, logical frameworks) as well as monitoring by conducting quality control throughout the entire project. Furthermore, it included FRA staff from different departments in order to ensure the cross-cutting nature of the event. Ultimately, an internal and external evaluation of the project were also undertaken immediately after the event, which will inform

the subsequent FRF in 2018.

159 The FRF focused on three topics: Immigration & Asylum; Information Society; and Discrimination.

160 Internal Ex-post evaluation report FRF.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 16: Website visits between 2012 and August 2017

150,000

100,000

50,000

-

Recruitment - plus vacancies Procurement - plus calls Publication - only publication pages Home page EN News & events - News/Events/Home page Caselaw references & charterpedia & home pages FRF Other

Third, a more indirect way to evaluate FRA’s dissemination relates to the number of times it is

mentioned in research publications. For example, since 2003 over 90 scientific publications have been directly written about FRA, which shows a considerable interest in the institution, in addition to

numerous articles relating to the Agency’s work. Taking violence against women as an example, FRA has

been quoted in 190 academic publications from 2014 to 2016 – which is only those articles and other academic publications that FRA has been made aware of. This includes different types of publications such as Bachelor and Master theses, academic journals, conference reports and books. Interestingly, there is a wide spread of academic disciplines using FRA’s outputs on VAW including anthropology, politics, law, medicine and psychology. Furthermore, FRA outputs are quoted in academic work published on European level, Member State level and even on a global level (for instance an article in a New Zealand journal made use of FRA outputs). In sum, this seems to suggest that FRA’s outputs are widely used in academia on VAW issues. FRA has made available the dataset from its Violence Against Women survey for researchers to use for the purpose of their own analysis. This is also being done for

other FRA surveys and serves to enhance the use of FRA’s own data in academic publications.

Ultimately, apart from FRA’s actual dissemination activities it is also important to assess how the latter

are perceived by stakeholders. In the external survey, most respondents (47) found that FRA’s communication and dissemination activities were to a moderate extent effective in helping to promote fundamental rights between 2013 and 2017. A considerably lower number (30) found that FRA was to a large extent promoting fundamental rights between 2013–2017.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Figure 17: External Survey: In your opinion, to what extent are the Agency’s communication and dissemination activities effective in helping to promote fundamental rights between 2013 and 2017? (N=94)

50

45

40

35

30

25 47

20

15 30

10

5 7

4

0 3 0

To a moderate To a large To a small No answer Do not know Not at all extent extent extent

Apart from the overall positive trends, some interviewed stakeholders argued that FRA could do even more to disseminate its findings. For example, some interviewees suggested that FRA should get the media more extensively involved as a ‘multiplier’ in reaching an even greater audience. However, it is important to note here that the Agency has been very successful in reaching out to the media with respect to some of its deliverables, such as the Violence Against Women survey. Other interviewees mentioned that it is important that the dissemination activities are targeted enough to reach the right stakeholders. This point, however, raises the question of whether FRA’s outputs should be presented mainly for policy-makers and a broad audience (and thus in very accessible language) or mainly for experts and academics (and thus in very technical language). While FRA attempts to match both expectations by delivering different types of outputs, interviewed stakeholders were often not sure

what the core target group is.

Furthermore, one interviewee mentioned that FRA needs to be very careful about which aspects of its mandate it would like to communicate to a wider audience. Fundamental rights (especially in relation to migration) can be easily politicised, and this could ultimately have even a negative effect for FRA. Apart from that, many stakeholders also took note that a key obstacle to a more effective dissemination

strategy is FRA’s limited budget.

To conclude, FRA outputs are generally perceived to be accessible, which is illustrated by a high number of downloaded materials in 2016. In 2016, FRA was the European Union agency with the highest number of orders with 13,152 copies of printed FRA publications ordered, showing the increasing

popularity of and demand for FRA publications. 161 In respect to FRA’s dissemination activities it has

been shown that over the years FRA has become increasingly visible in social media, and traditional engagement activities are attended by a wide stakeholder group. Nevertheless, many stakeholders still

feel that there is room for improvement and that more could be done to disseminate FRA’s outputs.

161 Annual Activity Report 2016

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

4.2.5. To what extent have the recommendations from the 2012 evaluation related to the effectiveness and impacts been implemented?

The purpose of this section is to assess the extent to which the recommendations on effectiveness of the 2012 evaluation have been implemented in the period from 2013 to 2017. The 2012 evaluation

made two key recommendations on effectiveness, each of which will be assessed in turn below.

Prioritisation of work

One recommendation made in the 2012 evaluation refers to the prioritisation of FRA’s activities. It was suggested that together with the Management Board and possibly other stakeholders, a thorough review of priorities should be undertaken. ‘The objective should be to ensure the available resources are used in the most effective and efficient way, which may mean a smaller number of projects, stakeholder focus or scope of activities. It will not be possible for the FRA to continue an approach where the Agency

tries to fulfil everybody’s expectations to the same extent.’ 162

In the strategic plan 2013 to 2017, the Mid-Term Review and the annual activity reports, it becomes clear that FRA is still working on numerous different projects and FRA still aims to satisfy the demands of various different stakeholders. In the Multi-Annual Framework 2013–2017, the strategic priorities in regard to all subject areas were formulated with a similar weight and required a diversity of activities to be carried out. Furthermore, the annual activity reports confirm that in regard to all topic areas similar efforts have been undertaken to achieve the targets. 163 For example, in all thematic areas advice to relevant stakeholders has been provided, data has been collected and methodologies to analyse data have been developed, relationships with key stakeholders have been established and awareness among

people affected by certain human rights violations has been raised.

Apart from FRA’s equal involvement in different subject areas, FRA also attempted to do justice to the needs of different stakeholders. This is reflected in the Strategic Priority 1 and 2 of the Strategic Plan for 2013–2017, which stressed that FRA shall be relevant to EU-level stakeholders and the Member States. In respect to both aspects, the annual activity reports from 2013 to 2017 illustrate that FRA has been

increasingly consulted by EU Institutions. 164 Furthermore, additional activities have been undertaken to

make the Agency more relevant for the Member States. 165

FRA’s engagement with multiple stakeholders and its equal involvement in many subject areas has been seen as a concern by some stakeholders. For instance, a few interviewees have argued that FRA’s

mandate is so broad that it is often difficult for FRA to do justice to all aspects of it.

As shown above, FRA is still involved to an equal extent in a wide range of activities. However, FRA’s broad mandate, as stipulated in its Founding Regulation, prevents it from disproportionally limiting its targets to some subject matters. Nonetheless, FRA has focused certain subject areas based on current

needs, particularly Roma integration and the rights of the child – the latter particularly in relation to the

162 2012 evaluation, p. 98.

163 For example, in all thematic areas advice has been provided to relevant stakeholders, data has been collected and methodologies to analyse

data have been established, relationships with key stakeholders have been established and awareness among people affected by

certain human rights violations has been raised.

164 In 2016 FRA was asked to provide six legal opinions.

165 For example, in 2015 National Parliamentary Focal Points were set up.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

migration crisis. 166 For example, in the thematic area of migration, FRA provided regular updates on the emerging fundamental rights issues in addition to the planned projects. 167 FRA has also increasingly

focused on creating synergies between different subject areas in order to more effectively tackle different priority areas. For example, FRA carried out research on the rights of unaccompanied asylumseeking children combining the thematic areas of the rights of the child and migration, as well as the thematic area of non-discrimination, by looking at the situation of LGBTI asylum seekers. 168 Most of the interviewees consulted in the framework of this study have felt that while dealing with a multitude of

different subject areas, FRA has been very successful in achieving its mandate.

Overall it can be concluded that FRA took the recommendations of the 2012 evaluation into consideration as far as its mandate allows it. At the same time, it has become more effective and

efficient in addressing all subject areas.

Ad-hoc requests

In the 2012 evaluation, it was also recommended that FRA should develop a strategy for meeting increasing demand for ad-hoc requests, in order to ensure that there is a good balance between responding to external requests and the pertinent needs for research on fundamental rights issues. This

recommendation is directly reflected in the Strategic Plan 2013 to 2017 where it was stated that

FRA will plan and allocate adequate human and financial resources to its advice function. It will assess the needed skills and clarify roles and responsibilities without, however, underestimating the importance of the research and data collection on which the advice is based. The creation of a FRA database of all its opinions, including recommendations contained in reports, formal opinions, and other public statements will allow FRA to keep track of its policy line, and ensure consistency and impact tracking over time.

FRA has implemented this aspect of the Strategic Plan by creating a section on its website containing all FRA opinions. 169 Furthermore, in 2017 a leaflet was produced explaining the procedure on how opinions can be requested and what they offer. 170 Apart from FRA’s efforts to raise awareness on opinions, it can be argued that FRA’s capacity to react to requests has increased. While seven opinions were published during FRA’s Multi-Annual Framework (MAF) 2007–2012, in the period from 2013 to 2017 10 requests have already been successfully answered. Six of those requests were made in 2016, and on top of that FRA has answered 21 informal requests from the European Parliament. This is partly the result of the FRA’s Director actively encouraging MEPs, especially from the LIBE committee, to make use of the FRA and its capacity to inform policy on the basis of its existing research and expertise. In addition, in interviews it has been mentioned that several informal requests are also made by the Council every year. This shows that FRA has sufficient capacity and flexibility to deal with requests alongside on-going

project work.

166 In 2012 FRA carried out two projects in the field of data protection (Redress mechanisms and their use and a handbook on EU data protection

case law) and in the following years it only carried out one project per year in this field.

167 Annual Activity Report 2016, p. 21.

168 See Annual Activity Report (2016), p. 17.

169 http://fra.europa.eu/en/publications-and-resources/opinions

170 http://www.google.at/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwjbqe23qcHWAhWlIJoKHUaoApcQFggn

MAA&url=http%3A%2F%2Ffra.europa.eu%2Fsites%2Fdefault%2Ffiles%2Ffra_uploads%2Ffra-2017-factsheetopinions_en.pdf&usg=AFQjCNFDjEXR3cIi3F2SrKSbGSZsc9D7Ag

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Added Value

The assessment of added value measures the benefits derived from EU interventions against what could have been achieved by Member States (and in this case the EU Institutions, bodies, offices and agencies) on their own (as per Art. 5 of the Treaty of the European Union – principle of subsidiarity). This criterion applies particularly in the case of the implementation of EU legislation, as well as interventions dealing with cross-border issues, cooperation, and promotion of best practice, as in the case of the FRA.

Important for all EU agencies, and perhaps even more so for decentralised agencies, is to be able to demonstrate the added value of their work. The added value of FRA and its interventions will in some ways be based on an overall judgement of the evaluation findings, but will also require examining stakeholder perceptions of the additional value that Agency-level intervention provides to the promotion and protection of fundamental human rights. As such, demonstrating added value involves canvassing stakeholders, both internal and external perceptions and opinions as well as undertaking an economic analysis to demonstrate the costs as well as any benefits that arise as a result of Agency-level intervention as opposed to intervention at an individual Member State-level and Institutions, bodies, offices and agencies level. Undertaking a cost-benefit

analysis of the Agency’s outputs will help to demonstrate the Agency’s efficiency (see section 4.5), but also demonstrate any

wider or societal benefit associated with the FRA’s activities. By comparing costs and benefits of Agency interve ntion versus intervention by individual Member State and EU Institutions, Bodies, offices and agencies, the generated net benefit (if any) is to be described as additional or added value.

4.3.1. What is the overall added value of the FRA?

The survey results indicate an important added value of FRA’s activities. In the survey involving FRA

internal staff, 72.73% of the respondents answered that FRA provides added value to the EU in the field

of fundamental rights ‘to a large extent’, while 18.18% of the respondents answered ‘to a moderate

extent’. Only one respondent said that FRA activities provided no added value at all, but the same

respondent also mentioned FRA’s surveys as a unique contribution to the protection a nd promotion of

fundamental rights in the EU – which would appear to contradict the respondent’s overall assessment.

Table 15: Added value of the FRA according to respondents in the internal survey (N=66)

Scale Answers Ratio

To a large extent 48 72.73% To a moderate extent 12 18.18% To a small extent 2 3.03% Not at all 1 1.52% Do not know 3 4.55% No answer 0 0%

In the two external surveys involving EU and non-EU actors, the question asked was about the unique contribution to the protection and promotion of fundamental rights (see EQ16) rather than added value specifically. However, also in these two surveys, the majority of respondents answered in a positive way.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

What is the additional value from the FRA outputs compared to what is done at MS level in

terms of providing assistance and expertise in fundamental rights issues?

What is the additional value from the FRA outputs compared to what is done at EU level by the EU Institutions, bodies, offices and agencies, in terms of providing assistance and

expertise in fundamental rights issues?

The above-mentioned sub-questions have been grouped together as the added value highlighted by the consulted stakeholders is similar at both national and EU level.

Overall, the views expressed regarding added value of FRA in the stakeholder interviews conducted are positive. There are a few exceptions; for example, only one interviewee from the CoE (and there were a number of other interviewees from the CoE) said that the FRA has no real additional value in general, and in the area of Roma integration in particular. A few survey responses indicate that the added value of the Agency might be less for the EU Member States than at the EU level. One difficulty that was highlighted is that FRA needs to stay independent and balance influence from various stakeholders, such

as academia, NGOs, and the Commission (Scientific Committee).

The sections below outline some of the main themes identified in the data collection phase and present a selection of views of the consulted stakeholders.

Research and comparative data

Most interviews conducted indicate that the FRA’s outputs provide added value compared to what is done in the field of fundamental rights both at the national level and the EU level. In particular, FRA collects and analyses data from across all EU Member States at one single moment in time. The comparative aspect of the research carried out is unique for the Agency and provides a basis for the work and policy development in the context of fundamental rights in the Member States, at the EU level, and for civil society. Interviewees believed that this EU-wide coverage distinguished FRA from other international organisations such as the OSCE or the Council of Europe, as well as from NGOs at national level. Interviewees at the national level mentioned that such research would not be possible to perform at the national level. For example, in fundamental rights areas where certain countries perform poorly they would be reluctant to gather this data. Also, the Member States might be too close to the problematic issues and miss the overview across Member States. There is a gap between what countries commit to and what happens on the ground, and this is something that FRA can investigate in an independent way and through the eyes of an observer. It was mentioned that the comparative data provided by FRA is sometimes the only reliable source of data on a specific topic and can fill gaps where national data is not available. Furthermore, the comparative data can indicate shortfalls at the EU level and, as a result, the Commission and European Parliament can recognise where the issues are located and make sure to focus on improvement activities in these areas. The data is also an important tool for advocacy and is used by many civil society organisations to justify a specific project initiative. One interviewee from the Council of Europe stated that the FRA is the only organisation collecting this type of data at European level, and that the Council of Europe relies heavily on this data, which according to the interviewee is a significant enough reason for the Agency to exist. This was highlighted in a recent publication for the European Parliament which stated that ‘Perhaps the most nuanced and relevant reporting at the European level is provided by the Fundamental Rights Agency, through both opinions

and analysis of data, but it is unclear how influential FRA reports and opinions are in terms of policy

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

change and there is a need for more up-to-date reporting on Islamophobia and anti-Semitism in

particular’. 171 It is important to note that this report was published prior to the Agency’s own release of

results from its ‘Second European Union Minorities and Discrimination Survey (EU-MIDIS II): Muslims – Selected findings’.

FRA provides added value in terms of gaining knowledge of ‘ No other organisation collects this data

what is happening on the ground in the area of fundamental in an independent way. The data

rights, according to an EC representative. This relates both to provided by NGOs are not always

surveys conducted by FRA as well as to the assistance provided independent. For me independent

in the hotspots. In the cases of assistance in the hotspots, FRA research is very important. ’

was instrumental in reporting on what was going on monthly. Member of the LIBE committee

This way, other agencies focused on the security aspect, while FRA ensured a holistic consideration of the situation, reporting on the fundamental rights.

According to DG HOME, the EU-wide overview and the capacity to look at the same issue across borders makes FRA’s output different to that of Member States. While the Commission has similar activities, they are more focused on legal issues while FRA’s activities are research-based. FRA is in a position to collect and disseminate information on policy and practices in the EU Member States that the EU Institutions might not be aware of, providing a comparative viewpoint that other institutions at national or EU level do not have (EC).

At a time when fundamental rights have been at the centre of political debate in Member States, the role of the FRA is even more important as a reminder of standards and rights. The backing of FRA in terms of research, standards, and supporting human rights initiatives is therefore of high significance, particularly in Member States experiencing limitations of rights and freedoms. Reports and handbooks are often provided in all EU languages, giving the user a product that is ready to use. However, the translation work might be moving to the Member States, according to FRA staff.

Independence

The role of FRA as an independent part of the larger international framework was mentioned as an added value by several interviewees. The independence of the Agency leads to trustworthy results, reliable and objective data. This might not always be the case for data gathered by NGOs or civil society as there is a possibility the specific agenda of the organisation is influencing the results. The external and objective perspective of FRA is of high importance both at the national and EU level. As an agency that is independent from, but still part of, the EU framework, it has an important critical voice. According to one interviewee at the international level, the independent role of the Agency could be strengthened even further.

The FRA has an important influence, ensuring the presence of the fundamental rights aspect in policymaking. The fact that the FRA is an EU Agency gives more weight to the opinions, and the high technical expertise and resources available provide an added value per se. Opinions of a well-established NGO might be similar; however, they might have less weight (civil society). It was said that a detailed and well-prepared opinion from FRA cannot be ignored by the European Parliament due to FRA’s level of expertise (civil society).

One interviewee at the international level underlined that the additional value of the Agency is that the

171 CEPS, Towards A Comprehensive EU Protection System for Minorities, 2017.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

EU has a separate independent institution on fundamental rights that can advise the institutions on upholding the European Charter of Fundamental Rights. The training on the Charter as an instrument, provided by the FRA, was mentioned as bringing additional

value by one interviewee at the national level. While they ‘ FRA is avoiding expressing its opinion, might be aware of the content, training on how to use it was which is very important to the objective said to be valuable for actors in the Member States. reputation of the Agency and its According to one interviewee (Advisory Panel), the Agency credibility. With FRA reports we are has been good at highlighting cases of contradictions confident that it’s absolutely correct, and between the Charter of Fundamental Rights and the position it interprets the law correctly. ’

of the EU. Representative of a national Ministry of Justice

Another added value is the fact that FRA is an institution which is solely and explicitly dedicated to fundamental rights. The overarching approach to fundamental rights across the EU is important. It was mentioned by one interviewee from the Advisory Panel that there is a lot of ignorance regarding the importance of fundamental rights both at Member State level/governments and among the general public. Consequently, FRA has an added value related to awareness raising.

Furthermore, FRA is valued for its objective reports that deliver opinions based on evidence. This enables the FRA to retain a good reputation and the credibility of an objective agency. According to one national-level interviewee, this is different from the UN, where recommendations are sometimes felt to be based on opinions and what has been brought to their attention.

The role of FRA – watchdog, monitoring or research

Based on the stakeholder consultation performed in the context of the evaluation, FRA provides an added

‘ Evidence-based human rights policy can value compared to what is done at national and EU level in

only occur when someone collects the terms of providing evidence-based assistance and expertise

information and the Agency is the only underpinned by large-scale research on fundamental rights

one in the EU that does. The CoE has to issues in the EU. As outlined above, added value can be found

rely on FRA data, which in itself would be in FRA’s research activities, the conclusions and opinions, the

significant enough reason for the Agency stakeholder engagement, and the independent nature of the

to exist .’ Agency, as well as in the monitoring role. While some of the

Member of the Management Board of CoE aspects mentioned as added value by stakeholders are not included in FRA’s mandate, e.g. its role as a watchdog, it can be

considered as an indirect effect of the important work and activities carried out by FRA regarding the protection of fundamental rights.

Some interviewees have mentioned the Agency’s role as a watchdog, indicating that this role should be expanded as no other institution has that purpose. One interviewee suggested that while the research activities are important, FRA should have a stronger role of monitoring in specific Member States that are weaker in the area of fundamental rights. However, there are different opinions on FRA’s role and mandate. A Council representative considered FRA rather as a think tank with policy advice, than a watchdog. One international-level interviewee mentioned that even though FRA does not have a direct mandate, people are now expecting the Agency to have a watchdog role, due to the standards that FRA is representing. Nevertheless, the same interviewee highlighted that he/she was not in favour of an extended mandate to include this role.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

According to the FRA Founding Regulation, Article 2, the ‘objective of the Agency shall be to provide the relevant institutions, bodies, offices and agencies of the Community and its Member States when implementing Community law with assistance and expertise relating to fundamental rights in order to support them when they take measures or formulate courses of action within their respective spheres of competence to fully respect fundamental rights.’ However, the FRA does not conduct systematic and

permanent monitoring.

Stakeholder engagement

FRA offers an important forum for discussion through the Fundamental Rights Platform (FRP) and other stakeholder meetings and events. It promotes a culture of fundamental rights and raises awareness. The role as a coordinating body bringing together all types of stakeholders in the field from all Member States, was mentioned as highly important. Even if the stakeholder engagement could be further

improved, the task is unique for FRA and this might be difficult to take over for another body in a

‘ FRA is critical in the EU’s credibility of constructive way. The Fundamental Rights Platform is looking into internal human rights issues considered an effective platform, currently in the phase of – it would not be the same if it was the adapting to a new strategy for cooperation between FRA and ECtHR. This is even more important now civil society institutions. While providing a platform for that the EU does not look to accede to stakeholders to meet and discuss, FRA gives a European angle the European Convention on Human to these debates. FRA can push the fundamental rights Rights in the foreseeable future. ’ concerns, while at the same time seeing other aspects of the

Member of the Management Board policy debate. According to a DG JUST representative, FRA facilitates the sharing of information and practices leading to

fruitful exchanges between Member States.

Another contribution of such forum, according to a Council representative, is seeing how the interests of the multiple stakeholders involved in fundamental rights issues are balanced. One interviewee mentioned that it is important for FRA to stay independent and balance the influence from the Commission, NGOs and academia. The NGOs are important users of FRA reports and FRA has more access to the NGOs than most other agencies. At the same time, the influence from Brussels is stronger

than the influence from NGOs, according to one interviewee (Scientific Committee).

Opinions, guidelines, and standards

With the overview of fundamental rights issues across the EU, the FRA has become an important source of information to be used for policy development, as well as a natural promoter for fundamental rights standards. The possibility for FRA to give policy recommendations or opinions at the EU level, specifying what the European Parliament and the Commission should do respectively represents an added value according to several interviewees. However, it was mentioned by an interviewee that, while in theory this role of the Agency provides for a strong added value, not much is being achieved in practice (takeup at national level). The selection process for the Director and the present evaluation were mentioned as opportunities for improvement. It was also suggested that specialised staff are needed to translate research into policy recommendations, while another interviewee said that the Agency staff have the skills and competence required. It was also said in the interviews that it would be of significant added value if the FRA could be more active in formulating policy recommendations (CoE). However, Member States governments and the Commission would not be in favour of this, according to one interviewee (CoE). The FRA Founding Regulation, Article 4, mentions that FRA shall ‘formulate and publish

conclusions and opinions on specific thematic topics, for the Union institutions and the Member States

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

when implementing Community law, either on its own initiative or at the request of the European Parliament, the Council or the Commission’.

FRA is seen by some as a guiding force for European standards in the area of fundamental rights. At the Member State level, one can refer to the work of the Agency when wanting to promote an area of fundamental rights either to civil society or to government institutions. The Agency can provide guidance across the EU Member States, providing an EU-level stance in a range of thematic areas, thus acting as the issuer of EU-level guidelines for the MS, according to one interviewee at the international level. A Frontex representative mentioned that FRA has an important mission in presenting differences across the EU and setting the minimum requirements/establishing a common baseline. According to one interviewee (multinational), FRA is open and willing to learn more about the human rights implications of big tech corporations (e.g. big data). This provides for an added value as it implies a stress on the role

of those companies in complying with fundamental rights.

What would be the most likely consequences of stopping FRA’s activities?

All interviewees that expressed themselves on this subject mentioned potential negative effects if FRA’s activities were terminated. The list below includes likely consequences that were mentioned.

 No other body or institution would be able to replace FRA and thus no institution would be dealing

with these important topics, such as discrimination (CoE). The specific and ad-hoc expertise of FRA and its staff, and the capacity of conducting comparative research in EU28 would be lost.

Consequently, this would lower the bar of research across EU in this field.

 It would imply the loss of an important interlocutor between the EU and civil society in the field of

fundamental rights, as well as a partner for collaboration and implementation (CoE). The platform for an EU-wide discussion on fundamental rights is not provided by any other EU institution. The

Agency’s meetings and platforms would be lost.

 The important role of an agency that translates the EU fundamental rights into practice and

interprets the Charter of Fundamental Rights would be lost. There would be no more independent advisory body and no research conducted in this field across EU28 in an independent way. A consequence of this would be less information about what is happening on the ground in the field

of fundamental rights and potentially also reduced fundamental rights standards over time.

 FRA is a reminder of the fact that the EU is based on human rights. The existence of FRA highlights

the importance of fundamental rights as a core issue which may have an impact on policy development.

 While other agencies deal with the operational issues, FRA’s critical and objective voice is needed

(FRA). According to one interviewee, the function of balancing out the other JHA agencies would be lost (Frontex).

 Research findings, policy recommendations, handbooks and other outputs needed to implement

fundamental rights at national level would be lost.

No positive consequences of stopping FRA’s activities were mentioned in the interviews conducted. In addition to the interview data, the same question was asked in the three different surveys. The survey responses largely confirm what was said in the interviews and indicate that stopping FRA’s activities would be a loss both at national and EU level. The lists below include negative consequences that were

mentioned by the survey respondents:

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

 Fundamental rights obligations and perspectives would more often be neglected and/or

disregarded in preparation of new directives or other instruments. Likewise, decision-making would be less informed. The lack of information and data would also result in limited opportunities to

react to societal challenges.

 The current situation and state of fundamental rights in the EU, with political extremism gaining

ground, was particularly mentioned and several respondents highlighted that for this reason FRA’s existence is even more important.

 Other agencies would have less support on how to address fundamental rights challenges in their

different policy areas.

 It would be a loss for the EU Institutions, NGOs and civil society, as well as for academia and

equality bodies. These actors currently use FRA’s data that has been gathered in an independent way and by a unique combination of legal and social research.

 FRA has managed to reach out to the general public and has evolved over time using the latest

technologies available. The awareness-raising effects of FRA would be lost.

The survey responses also included a small number of comments highlighting potential positive consequences of stopping the FRA’s activities. These comments are listed below:

 Minuscule savings to the EU budget. However, this would not comparable to the value lost (internal

survey respondent);

 While most survey answers were straightforward, a few respondents expressed uncertainty due to

the fact that the impact of the FRA varies very much from country to country, and depending on the subject.

On the basis of the stakeholder perceptions accounted for in this section, it seems as if a discontinuation of the FRA’s activities would mostly result in negative consequences. According to the majority of stakeholders consulted in the interviews and the survey, it would not be advisable to stop these activities. Several negative consequences were mentioned. Furthermore, it is important to note that these consequences might also have long-term effects related to the actual protection of fundamental

rights, policy development, and research in the field across the EU.

Confidence in FRA output

The data collection carried out in the context of this evaluation indicates that relevant stakeholders are confident of the FRA’s outputs. No doubts or lack of confidence were expressed regarding the Agency’s outputs. It was mentioned that the comparative data gathered by FRA from across all EU MS cannot be done in the same objective and independent way by the MS themselves, and such data is not gathered by the Commission in a systematic way. Furthermore, the fact that FRA has a specific objective and resources set aside for these activities enables the research to be done in a structured way and with robust outputs. Funds and resources for research were mentioned in the interviews as an added value of the Agency (e.g. Microsoft). This aspect makes FRA different from other organisations and bodies, as well as to NGOs. The available resources also enable the possibility of repeating research over time, as in the case of EU MIDIS. This is valuable and ensures continuity. It should be mentioned, however, that

some interviewees said that the FRA’s budget is limited (EP).

Interviewees have indicated that the models and methodologies for the FRA’s data collection are replicated by OSCE and the Council of Europe for surveys conducted outside of Europe, such as the

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Violence Against Women survey, which has been replicated by OSCE in 10 non-EU countries. In addition, one CoE representative said that the CoE relies on the data resulting from the FRA’s activities. According to one interviewee, the FRA has developed an authoritative source and voice in the context of fundamental rights at the EU level in recent years, and it has a capacity to provide comparative data that is unique for the Agency. While other organisations have a more political role, FRA has a particular

mandate for advice and research which is key, and complementary to other EU bodies.

The Council of the EU mentioned that FRA provides a good and reliable source of information, figures and data. The value of this relates to policy advice and development, as well as to awareness raising and debates at national level. In addition to national and EU bodies using FRA outputs, NGOs and civil society are important users of the data. Furthermore, the possibilities of interaction with the Agency through

the FRP and other stakeholder meetings may further confirm confidence in the research output.

4.3.2. What has been the unique contribution of the Agency to the promotion and protection of fundamental rights in the EU?

Are the objectives and outputs of the FRA unique (i.e. no overlapping outputs with other

similar organisations at EU, national or international level)?

In the survey, looking at EU and non-EU actors combined, 52% of the respondents believed that the Agency made a unique contribution to the promotion and protection of fundamental rights in the EU between 2013 and 2017 to a large extent, while 34% of the respondents answered ‘to a moderate extent’ (see Table 16). The total number of respondents for this question was 91, of which external non

EU survey was 77.

Table 16: Unique contribution of the FRA to the promotion and protection of fundamental rights in the EU (external survey, EU and non-EU actors)

Answers Ratio

To a large extent 47 52%

To a moderate extent 31 34%

To a small extent 4 4%

Not at all 1 1%

Do not know 3 3%

No answer 5 6%

In addition to the survey, the interviews conducted highlight the contribution of the FRA in the field of fundamental rights. The aspects mentioned are similar to the answers provided regarding the overall

added value of the Agency (section 4.3.1) and are further outlined below.

The EU-wide comparative research was highlighted by the interviewees at various points as something unique for the FRA. Furthermore, the independent nature of the Agency – separate from both Member States and other EU bodies – differs from other organisations in the field. Duplications of outputs may occur, according to one interviewee (EC). However, as the research conducted is considered unique, the risk of duplications should be limited. The objectivity of the FRA in gathering data on fundamental rights in the EU Member States is of high importance, as the perspective from an external observer may be

more critical than an actor that is close to the problematic being investigated. At the same time, the fact

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

that the FRA is an EU agency implies that opinions emanating from the Agency weigh more than e.g. opinions and research from civil society.

It was also mentioned that the FRA has an important role in providing a forum for discussions on fundamental rights involving relevant stakeholders from across the EU Member States. This includes the Fundamental Rights Platform and other stakeholder engagement activities, and particularly the extent/range of stakeholders involved in such activities that is unique for the FRA. Regarding awareness raising and influence on other actors in the field of fundamental rights, the fact that FRA is a body that has fundamental rights at the core of all activities is an added value per se, and a unique aspect among

the EU bodies.

A representative from the Scientific Committee mentioned that the Agency has an innovative aspect in its research approach, which is bottom-up, and not only top-down. In addition, this is combined with a legal approach. According to the interviewee, the FRA is unique and different to the UN Commission on

Human Rights and the CoE cases for this reason.

The survey responses to the open questions regarding the FRA’s unique contribution largely confirm the interviewees’ opinions. It should be noted that all three surveys conducted with FRA internal staff, external non-EU actors, and external EU actors indicate the same activities as unique. Examples

mentioned in the survey responses include the following:

 Reminding decision-makers of their obligations in ensuring fundamental rights, guiding them

through human rights standards, and monitoring the situation on the ground;

 Provision of comparative data across all EU MS, including large-scale surveys (e.g. VAW, LGBT, EU-

MIDIS, Roma and Anti-Semitism) and handbooks;

 Provision of evidence-based opinions to the Commission and the European Parliament, policy

recommendations and guidance. FRA opinions are increasingly requested;

 Provision of a forum for debates and discussions. The Fundamental Rights Forum organised in 2016

was highlighted;

 FRA as an internal critical voice which indicates both challenges and good practices for EU activities;

 FRA’s presence both at EU level and at the national level in parallel;

 Deployment of staff to refugee hotspots in Italy and Greece;

 Awareness raising and increased coordination on fundamental rights in the EU;

 One-stop-shop approach for research, analysis, publication, dissemination and awareness raising

relative to fundamental rights;

 Thematic areas that were mentioned specifically include refugee protection, Roma integration, data

protection, hate crime, and labour exploitation.

Based on the stakeholder perceptions gathered through the survey and the interviews, it is clear that the FRA is contributing importantly, and in a unique way, to the promotion and protection of

fundamental rights in the EU.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Assessing Coordination and Coherence

When assessing coherence, it is important to check how different components of the intervention operate together and whether the intervention is in line with and does not contradict other policies and priorities at EU and national level. This criterion essentially examines the coherence of interventions with the overall strategic objectives of the EU (Europe 2020, EC political guidelines, European Council strategic guidelines, etc.), the synergies and complementarities with other sectoral interventions.

Coordination is thereby an important element of coherence, essentially assessing the way in which coherence across the sector takes place. As such, it is proposed that the Team will examine the level and extent of cooperation between the Agency and its stakeholders in order to meet the Agency’s objec tives.

4.4.1. To what extent is the Agency acting in close cooperation with the CoE and UN to

avoid duplication and in order to ensure complementarity?

Cooperation with the CoE and UN

Since the establishment of the FRA in 2007, collaboration with other international institutions in the field of fundamental rights has been central. Various actors are active in this field both at the European and international level, and it is thus crucial to communicate and collaborate closely to work effectively, avoid duplication and ensure complementarity.

One of the main actors at the European level is the Council of Europe (CoE). All Member States of the EU are also members of the Council of Europe and collaboration between the two organisations has been

considered of great importance from the start. The FRA’s Founding Regulation (Art. 9) establishes that

the Agency shall ‘coordinate its activities with those of the Council of Europe’. 172 The cooperation is reinforced through an agreement between the FRA and the Council of Europe from 2008. 173 Shared priorities related to human rights and fundamental freedoms are at the core of this collaboration.

Regarding the international level, the United Nations (UN) is the key actor. While the UN has a widereaching mandate and a global scope, there are several areas where cooperation between the UN bodies and the FRA has clear potential. There is a long-standing collaboration with inter alia UNODC, UNICEF, UNECE, ILO, UNHCR, IOM, UNDP 174 and OHCHR in areas such as migration, child protection, violence against women, and the situation of Roma people 175 .

To what extent is the Agency engaging in such cooperation activities, and since when? Has there been a change in cooperation activities over the last 5 years?

The FRA Founding Regulation mentions collaboration between the Agency and both the CoE and the UN, as well as with other international organisations, in order to carry out its tasks. The data collection in the context of this study indicates that there is a robust structure in place for collaboration, as well as informal channels used for exchange of information.

172 Council regulation (EC) No 268/2007 i 173 Cooperation agreement FRA/CoE, 2008 174 Protocol for cooperation FRA/UNDP, 2011 175 http://fra.europa.eu/en/cooperation/osce-un-international-organisations

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

As mentioned above, Article 9 of the Founding Regulation defines the cooperation with the CoE. It introduces consultations during the preparation of the FRA’s AWPs and the annual reports on

fundamental rights issues. 176 In this way, priorities, activities and findings of the CoE are taken into

consideration in FRA’s key documents. Furthermore, the Regulation establishes the appointment of a contact person in each organisation responsible for dealing with matters related to this cooperation.

The objective of the cooperation agreement from 2008 is to avoid duplication and ensure complementarity and added value to the work of both organisations. Over the years, this cooperation

has further evolved towards an improved coordination of activities and synergies. The focus of the cooperation includes the following 177 :

 developing joint projects in areas of mutual concern;

 engaging in dialogue with stakeholders to improve the situation of fundamental rights in Europe;

 coordinating communication activities to increase awareness regarding fundamental rights;

 informing each other on the results of activities of each organisation; and

 exchanging data and consulting each other at operational level.

Moreover, one person from the CoE sits on the FRA Management Board and on the FRA Executive Board. This person is appointed by the CoE to participate in the decision-making of the Management

Board, which adopts the FRA’s AWP and the annual reports, as well as appointing members to the FRA

Scientific Committee. 178 Operational collaboration and joint projects between the two organisations

include the work on handbooks in different areas, such as data protection; asylum, borders and

migration; and the rights of the child. 179 These books were mentioned in the interviews conducted as

successful examples of collaboration.

Regarding other international collaboration, the FRA Founding Regulation, Art 8.2, establishes cooperation with the Organisation for Security and Cooperation in Europe (OSCE), especially the Office

for Democratic Institutions and Human Rights (ODIHR), the United Nations and other international organisations. 180 A protocol was signed with the UNDP in 2011 establishing collaboration in the areas of data collection and research; networking and common events; communication and awareness-raising activities; and capacity development. In addition, an exchange of information with the OHCHR takes

place inter alia during the drafting phase of the FRA’s AWP, and FRA has collaborated on several surveys with e.g. UNDP 181 and UNECE 182 . Cooperation also takes place in different thematic areas; for example

the Agency coordinates with the UNHCR in the area of migration and asylum. 183 One international-level interviewee mentioned that ODIHR would like to have closer cooperation with FRA; however, the

Agency has appeared reluctant to collaborate at the level suggested by ODIHR at different occasions. Despite this, there is a small day-to-day engagement between the two organisations and ODIHR

considers FRA an important collaboration partner.

176 Article 9, Council regulation (EC) No 268/2007 i

177 Overview of the cooperation between the European Union Agency for Fundamental Rights and the Council of Europe (2013–2014)

178 http://fra.europa.eu/en/cooperation/council-of-europe

179 Overview of the cooperation between the European Union Agency for Fundamental Rights and the Council of Europe (2013–2014)

180 Article 8, FRA Founding Regulation

181 The situation of Roma in 11 EU Member States – Survey results at a glance

182 Violence Against Women

183 Annual Work Programme 2016

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

The surveys conducted in the context of this study with both internal and external actors suggest that there is a well-established cooperation between FRA and the Council of Europe (both regional organisations active on the European soil), as well as with the UN bodies. Figure 18 illustrates the responses from the internal survey with FRA staff.

Figure 18: Cooperation between FRA and the UN and CoE (internal survey, N=66)

Is there adequate resource allocation for cooperation activities?

In order to assess the FRA against other Agencies it is important to look at the FRA’s budget, in which

cooperation activities are included in the ‘cross-cutting activities’. These activities include communication and awareness raising, consulting mechanism and bodies of the Agency, and research and data collection. It can be noted that expenditures in this area seem to have been stable over the past five years, with an important increase planned for 2017. 184 However, cross-cutting activities also include activities undertaken beyond the planned projects such as the deployment of FRA staff to provide support to authorities at refugee hotspots. Therefore, the increase of the budget might be related to these additional efforts. The increase is also reflected in the operational human resources, where the number of staff for cross-cutting activities has increased by 13 in the planning for 2017.

Table 17: Financial resources for cross-cutting activities

2013 2013 185 2014 186 2014 2015 187 2016

Planned Actualised Planned Actualised Planned 2015 Planned 2016

2017

operational Direct operational Direct operational Actualised Actualised

Planned operational

expenditure expenditure expenditure expenditure expenditure Direct

operational

expenditure Direct expenditure

Cross-cutting

activities €1,880,000 N/A €1,635,000 €251,956 €1,275,000 €297,743 €1,435,000 €2,580,000

184 Annual Activity Report 2014, 2015

185 Annual Accounts of the FRA for the Financial year ended 31 December 2013

186 Annual Activity Report 2014, European Union Agency for Fundamental Rights, 2015

187 Annual Activity Report 2015, European Union Agency for Fundamental Rights, 2016

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

2013 2013 185 2014 186 2014 2015 187

Planned Actualised Planned Actualised Planned 2015

2016 2016 2017

operational Direct operational Direct operational Actualised

Planned Actualised Planned

operational

expenditure expenditure expenditure expenditure expenditure Direct

operational

expenditure Direct expenditure

Research and Included in Included in Included in Included in Included in data collection crosscross- crosscross- crosscutting €719,864 cutting €226,321 cutting €256,379 cutting cutting activities activities activities activities activities expenditure expenditure expenditure expenditure expenditure Communication Included in Included in Included in Included in Included in and awareness crosscross- crosscross- crossraising cutting €1,097,815 cutting €1,026,068 cutting €915,047 cutting €4,100,890 cutting activities activities activities activities activities expenditure expenditure expenditure expenditure expenditure Bodies of the Included in Agency cross€435,000 €252,175 €475,000 N/A €490,000 N/A €365,000 €914,144 cutting activities expenditure Consulting

mechanisms Included in Included in Included in Included in

Included in

bodies of bodies of bodies of crossthe Agency €211,897 the Agency €190,535 the Agency €162,726

bodies of

the Agency €188,670 cutting

expenditure expenditure expenditure expenditure activities expenditure

The stakeholder interviews and surveys provide limited information on this question. However, one interviewee said that the cooperation activities are adequately budgeted and resourced. Furthermore, one survey respondent mentioned that the cooperation at the EU level has improved over the past five years thanks to, inter alia, increased resources spent in this area. Stakeholder relations officers and special liaisons were mentioned in this context.

To what extent are the activities proving effective in avoiding duplication and ensuring complementarity?

In general, cooperation activities with CoE and the UN are considered positive and effective according to most interviewees, and some have indicated that the cooperation has improved over time. Interesting stakeholder meetings as well as the FRA taking the initiative with other international organisations were mentioned as positive factors, in particular with the UN and the Council of Europe. Also,

operational cooperation and joint publications such as the Handbooks 188 were mentioned as good

examples. Another successful example of collaboration is the Violence Against Women survey that was conducted by the Agency. This survey is being replicated by the OSCE in 10 non-EU Member States, using the same questionnaire and methodology, and with the FRA as part of the Steering Committee for the replication.

While the overall perception seems positive, only a few interviewees and survey respondents have a more negative picture of the cooperation.

While the total number of EU respondents to the survey (14) is too limited to be duly considered, it indicates a positive trend and no large overlaps. External non-EU actors have responded in a similar way to the online survey. Most non-EU survey respondents (out of 76) believe that there are overlaps to a small or moderate extent, or not at all, while few respondents have indicated that there are large

188 The handbooks are designed to assist legal practitioners at national and European level, including lawyers, judges, prosecutors, border

guards, immigration officials and others working with national authorities, as well as non-governmental organisations and other

bodies that may be confronted with legal questions in any of the areas the handbook sets out to cover.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

overlaps. According to the graph in Figure 19, the large overlaps would be with the CoE rather than with the UN.

Figure 19: Overlaps between FRA and the CoE or the UN (External non-EU actors, N=76)

Finally, the internal survey with FRA staff suggests that the cooperation in place is effective to a large or moderate extent, and several respondents reported that there has been an improvement in recent

years (see Figure 20). Some said this might be connected to the increased recognition of the Agency’s work.

Figure 20: Improved level of cooperation over time (2013–2017), (internal survey, N=66)

A strong preparedness and will to cooperate and share information in both the FRA and the Council of Europe were mentioned by FRA staff. While the external survey pointed out that cooperation with the Council of Europe was functioning well, suggestions were made as to why and how it could be further improved. For example, one national-level interviewee said that more cooperation projects would be beneficial, but that there is competition between the two organisations. Similarly, one Commission

interviewee mentioned an ‘institutional jealousy’ between the CoE and FRA. While various divisions of

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

the Agency and the Council of Europe meet, it was mentioned that such cooperation activities could be more regular.

In general, interviewees reported that there are no or few issues of duplication regarding the Council of Europe or the UN bodies. While one interviewee suggested that there is some overlap between the Council of Europe and FRA, it is important to have FRA’s EU perspective which might be different to the

perspective of the Council of Europe.

Regarding potential overlaps, it was highlighted that due to the different structures of the organisations, they complement each other rather than duplicate the work. Examples mentioned included the work on European standards in the area of domestic violence combining judgements of the human rights court by the CoE and national practices by FRA, joint publications, and the handbooks. In this way, the risk of overlapping is well managed. One interviewee welcomed more concrete collaboration projects with the CoE. Other interviewees said that there have been improvements in recent years. Regarding the UN, it

was said that this organisation has a much larger mandate and geographical scope.

4.4.2. To what extent is the Agency ensuring appropriate coordination with relevant stakeholders to foster synergies and avoid duplication?

Appropriate coordination with relevant stakeholders is key for the FRA’s work. Numerous stakeholders at the national, European and international level are relevant for the work of FRA and include EU Institutions and bodies, NGOs and civil society institutions at national level in the field of human rights.

Such cooperation is established in Article 7-10 in the FRA Founding Regulation.

To what extent is the Agency ensuring appropriate coordination with relevant EU Institutions, bodies, offices and agencies to foster synergies and avoid duplication, and since

when? Has there been a change in cooperation activities over the last 5 years?

FRA has a close and regular cooperation with the European Parliament, the European Commission and the Council of the EU. Furthermore, the Council defines the FRA’s MAF every five years after a proposal from the Commission and consultation with the European Parliament. The FRA Management Board and Executive Board include Commission representatives. Regarding other agencies, FRA cooperates in

particular with Eurofound, EIGE, Frontex, Eurojust, EASO, Europol, EAHC, eu-LISA and CEPOL. 189 During

the period March 2014 to February 2015, the FRA chaired the EU Agencies Network, with the aim to improve the coordination, information exchange and agreement of common position of issues of shared interest among the decentralised agencies. 190 In 2016, FRA also coordinated the cooperation of EU agencies in the context of JHA (Justice and Home Affairs inter-agency cooperation). This group aims to coordinate operational work and to explore synergies in the areas of freedom, security and justice. 191 In addition, the FRA undertakes human rights training with specific stakeholders and provides expert input in the development of human rights training of other EU agencies (e.g. Frontex and CEPOL). 192 In recent years, cooperation agreements were signed with eu-LISA (2016), Eurojust (2014), and EASO (2013). Cooperation agreements with Frontex, Eurofound and EIGE were signed before this. Finally, the abovementioned

agencies are consulted on an annual basis on the activities and outputs planned by FRA.

189 Annual Work Programme 2016

190 Annual Activity Report 2015

191 Annual Work Programme 2016

192 Annual Work Programme 2016

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Regarding EIGE, the two agencies have cooperated on the use of the Violence Against Women survey’s

results. EIGE was given privileged first access to FRA’s survey dataset, which was used as the basis of

EIGE’s data input for the ‘violence’ domain in the EIGE gender equality index. FRA also provided EIGE

with the analysis of its survey data on Roma on the basis of gender, which EIGE used for its own

publication work. Finally, EIGE is invited to participate in the FRA’s Management Board meetings and

has been present on various occasions. However, so far FRA has not been invited to participate in EIGE’s Management Board meetings. The need for strategic cooperation between the two Agencies was mentioned in the interviews. While they work well together, both organisations would benefit from clarifying the mandates of the respective agencies, according to one interviewee. However, the few interviews that referred to this highlighted that a merger would not be beneficial as this would lead to both reduced visibility and resources for the successor Agency.

The staff survey indicated a high level of cooperation between the FRA and the EU Institutions and bodies (see Figure 21), in particular with the European Commission.

Figure 21: Extent of cooperation between FRA and EU actors (internal survey, N=66)

While the survey conducted with external EU actors had a too low response rate to be duly considered, the responses received nevertheless indicate strong cooperation.

A good relationship with EU actors in the field of fundamental rights was reported by most interviewees. One survey respondent said that FRA has become a ‘reliable service provider’ to the EC, EP, Frontex, eu - LISA and EASO, as the Agency is well linked to the institutions and able to provide relevant quality input at the right time. ‘ We greatly benefit from the FRA’s

No specific overlaps were reported, even though one work – it has a well-developed system

interviewee mentioned that there should be more […] we can get access and make good

coordination regarding activities and outputs with other use of their outputs. ’ stakeholders, in particular with Eurofound. Frontex stakeholder

Collaboration with the Council presidency was highlighted. Informal contacts between FRA and the country of the presidency are reported to be positive for both parts to identify areas of common interest. The informal exchange between FRA and the Council was also reported by another interviewee, mentioning a ‘well established cooperation’ with a ‘quick

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

response’. Furthermore, training provided by FRA on how to use the Charter of Fundamental Rights in the work of the Council was mentioned as a positive example, as well as the organisation of conferences resulting in interesting discussions (e.g. concerning the VAW survey). This was showcased during the

Maltese presidency when a number of symbolic and impactful events took place. Prime amongst them was the invitation of the Agency’s Director to participate in the informal Justice and Home Affairs

Council in Valletta (see above). More symbolic aspects included the participation of the President of Malta at the launch of the Agency’s annual Fundamental Rights Report in June 2017. Furthermore, an

important number of meetings and conference took place over the length of the presidency.

Regarding collaboration with relevant Directorate Generals of the European Commission, technical exchanges as well as informative exchanges on different projects, and consultations on work programmes and annual reports were mentioned. While the FRA has a close collaboration with DG JUST,

it was reported that other relevant Directorate Generals are consulted as well and the Agency is invited to participate in a wide range of inter-service consultations and steering groups. For example, DG REGIO

requested FRA to contribute to the training of national management authorities of European Structural and Investment Funds on fundamental rights compliance. Collaboration between the FRA and DG HOME around the annual platform with civil society on migration was mentioned as a positive example

enabling exchange of experience and the possibility to participate in each other’s fora. Another interviewee highlighted the existence of informal cooperation networks between FRA and the Commission Directorate Generals, and it was said that the availability of FRA colleagues in informal

exchanges is much appreciated. It was mentioned that the European Parliament is in close contact with FRA with ad-hoc requests that are much appreciated.

Concerning FRA’s collaboration with other EU agencies, the role of FRA in providing practical advice on the ground was mentioned as positive. FRA serves as an independent and impartial actor that can

support in the observation of protection of fundamental rights. In particular, the presence of FRA advisers together with Frontex and EASO at the hotspots was indicated as a successful example where

the agencies make a difference in practice. In relation to this, it was said that such work increases the credibility of the Agency with other stakeholders, and creates trust with other EU Institutions and governments in the EU Member States. In addition, one Commission interviewee said that FRA

leverages the good cooperation with Frontex and EASO to achieve impact in these areas. Finally, it was said that the close cooperation with other agencies is crucial as a connection between FRA and decisionmakers.

FRA and the EEA and Norway Grants

FRA cooperates with the EEA and Norway Grants with a view to complement each other’s work and enable public authorities and civil society to tackle fundamental rights issues in EU Member States. In December 2016, an administrative cooperation arrangement was signed between the Financial Mechanism Office and the European Union Agency for Fundamental Rights with the aim of strengthening synergies and coordination on strategic, programme and project levels.

The cooperation between FRA and the EEA and Norway Grants focuses particularly on the promotion of Roma inclusion, supporting responses to hate crime by public authorities and civil society, access to justice and follow-up to FRA’s survey on gender-based violence. In practice, the Grants were a key partner at numerous FRA events such as the Fundamental Rights Forum, the Fundamental Rights Conference ‘Combating Hate Crime in the EU’. In relation to Roma integration, FRA has supported the implementation of SocioRoMap – an EEA/Norway Grants-supported project in Romania. Furthermore, in 2016 the Grants provided financial support to the conference ‘Ensuring cross-border justice for all in the

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

EU: sharing practices and experiences from the ground’, which was organised by FRA together with the Slovak Presidency of the Council of the EU. The conference brought together over 100 representatives from international and EU Institutions, national governments, the judiciary and other backgrounds.

To what extent is the Agency acting in close cooperation with non-governmental organisations and with institutions of civil society at the national level, and since when? Has

there been a change in cooperation activities over the last 5 years?

The Fundamental Rights Platform (FRP) was established by FRA’s Founding Regulation, with the aim of working closely with NGOs and institutions of civil society. According to the Regulation, such

cooperation shall enable information exchange and pooling of knowledge. 193 Furthermore, civil society

organisations interact with FRA through projects and activities of the Agency. FRA collaborates with inter alia social partners, professional groups, academics, practitioners, NGOs and experts. 194 Methods

of cooperation include meetings, consultations, communication on outcomes and recommendations of civil society conferences, and capacity building. This cooperation enables civil society to feed into FRA’s

programming documents and annual reports on fundamental rights. 195 In 2017, the FRP is relaunched under a new strategy aiming for closer cooperation with a broader variety of civil society organisations

through an enhanced thematic engagement. 196 The enhanced platform, which currently brings together 500 organisations, will be further shaped during the second Fundamental Rights Forum in September

2018. Interviewees indicated that this cooperation could both be deepened and enlarged, reaching out to more stakeholders.

The FRP and collaboration between FRA and civil society were mentioned as positive and important by various interviewees. Furthermore, the survey with internal actors indicated strong collaboration. It

was said that FRA often refers to civil society in their work and presentations, and also provides possibilities for the NGOs to gather and exchange knowledge and information. Interviewees reported that it is important for FRA to include both civil society and international organisations in their work. It

was also said that the cooperation has been improving significantly over the years and that the new strategy, mentioned above, will further improve the activities. According to these internal interviewees,

the new structure for cooperation may be less cumbersome, enabling easier registration for new organisations. This might open up the platform to other, less classical, NGOs. It was mentioned that the platform could be more operational on certain thematic issues and organised through working groups,

similar to cooperation with the national-level institutions. The Agency is in contact with the most relevant people in civil society; however, the interactions could be more intensive. One interviewee said

that increased ownership would lead to improved input from civil society.

NGOs are well represented in the FRP while universities and trade unions are less well represented. Regarding the trade unions, this might be due to the fact that they have other ways of influencing at the

EU level, and/or lack of interest – though they are engaging with FRA through expert meetings relating to the development of FRA’s research on severe forms of labour exploitation. One issue mentioned was

that contradictory opinions may take place among the various stakeholders involved in the FPR. The

193 Council regulation (EC) No 268/2007 i

194 Annual Work Programme 2016

195 Fundamental Rights Platform, Terms of reference, 2017

196 Terms of Reference of the FRP, 2017, FRA website

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

organisations might also be competitors for resources. This broad audience of FRA represents a challenge, according to some interviewees. One interviewee said that FRA could be clearer about the

purpose of the stakeholder engagement. FRA also engages with civil society through questionnaires to gather input and when communicating FRA reports and work. This helps build stronger partnership.

Indeed, communication with NGOs and organisations at national level results in further dissemination of results through a snowball effect. It was mentioned that at the EU level, NGO networks are well aware

of FRA’s work, while this might not always be the case at the national level. In this context, it was mentioned by one interviewee that the national members of the FRA Management Board have a

responsibility here to communicate on FRA’s activities at the national level. One interviewee said that there might be hostility towards FRA Management Board members in some Member States as some Member States do not agree with all of FRA’s activities. While the members of the Management Board

should be independent, it was indicated that some might be influenced by their governments.

The survey and interviews also mentioned cooperation activities with OHCHR, ODIHR, the national Roma contact points, EU representations and offices of international organisations in Member States, as

well as new cooperation agreements with the Norway and EEA grants in 2016. Also, increasing collaboration with national statistics offices was indicated.

Is the resource allocation adequate for the coordination activities?

Is the resource allocation proportionate?

How does the resource allocation compare to other decentralised agencies?

The three questions above have been grouped together.

As mentioned above, coordination activities are included under cross-cutting activities in the FRA’s budget. While the budget has been stable over the past few years, an important increase can be noted

in the planned budget for 2017. However, cross-cutting activities also include activities undertaken beyond the planned projects such as the deployment of FRA staff to migrant hotspots to provide

support, and the increase of the budget might be related to these efforts.

Under cross-cutting activities, one project focuses on cooperation with EU Institutions, agencies and bodies, while another project is dedicated to the FRP. Looking at the budgets allocated to these projects

from 2014 to 2016, it can be concluded that it remained rather stable, with the exception of 2016 when the budget for FRP was cut by more than 50%.

Table 18: Budget dedicated to cooperation activities (2014–2016) 197

EU Institutions, agencies and bodies Fundamental Rights Platform 2014 2015 2016 2014 2015 2016 €105,000 €90,000 €105,000 €140,000 €140,000 €50,000

There does not seem to be an overall agreement among stakeholders on whether resources are allocated sufficiently to meet the needs of stakeholder groups. One interviewee stated that the

allocation of FRA resources for cooperation activities is not sufficient to meet the needs of the relevant stakeholder groups, while another interviewee said that FRA seems to have sufficient resources for the

activities. It can be noted that the first interviewee is an NGO representative of the FRP Advisory Panel, while the second interviewee is from the Council and thus represents another perspective. One survey

197 Annual Work Programme, 2015, 2016

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

respondent mentioned that the FRA has improved significantly in terms of relations with key stakeholders at EU and national level. Increased resources are reported to be one of the reasons for this

improvement (e.g. specific stakeholder relations officers).

4.4.3. To what extent are the procedures to ensure this coordination and cooperation effective to ensure the Agency’s activities are coherent with the policies and activities of its stakeholders?

The sections above outlined the procedures and activities in place to ensure coordination and cooperation between FRA and key stakeholders at the national, EU and international level. The objectives of such procedures are to avoid duplication, foster synergies and ensure that policies and priorities across organisations do not contradict each other. Effectiveness in this case should thus

consider the degree of achievement of this objective.

Based on the data collection, the procedures in place to ensure coordination and cooperation are considered effective. Relevant stakeholders in the field of fundamental rights at all levels are accounted for in these procedures. Furthermore, the formal structures of collaboration are complemented by informal channels of dialogue between relevant actors. Some consulted stakeholders have mentioned a

need for further coordination, and the existence of overlaps.

Regarding informal channels of collaboration, several stakeholders noted this as a positive way of working. However, one interviewee highlighted that there are also issues related to this as the Agency may become too dependent on the people in the organisations and the existing relationships of the people working there. While individual openness is important, the collaboration should not be dependent on personal relations. To further improve work with the EC, it was suggested that a yearly meeting be organised with the responsible people in one area from both FRA and the EC. This could also be done more regularly where needed. The Fundamental Rights Platform is good for this regarding civil

society; however, only one or two people from the EC take part in this work.

Several interviewees mentioned the Agency’s efforts on accessibility and participation practices. It was said that there is room for civil society to be involved in the communication of research findings, and to respond to the initial findings and research design. One interviewee said that FRA’s collaboration with civil society is ‘one of the best organised dialogues’ that other organisations can learn from. However, it was also mentioned that the work needs to be focused and that FRA’s mandate is too broad. As a consequence, the cooperation does not always result in something concrete. Due to the broad mandate, there might be contradictions between organisations that are part of the collaboration. This might potentially affect the results of the collaboration. One interviewee mentioned that cooperation could be expanded to the local and regional level, particularly with the European cities, creating a platform that would be similar to Eurocities. Another interviewee highlighted the lack of cooperation with larger NGOs, saying that they are often absent from the Agency’s events. It was reported that FRA has more access to NGOs than most other EU agencies, while NGOs also use FRA and FRA’s reports regularly. However, one interviewee (Scientific Committee) pointed out that the scope and direction of FRA’s work is more likely to be influenced by the EU Institutions than by the NGOs. The importance of balancing between the views and opinions of the Commission, academia and NGOs was highlighted. It was also noted that some members of the network are very active while some are less active. The quality of the

network members is crucial for effectiveness.

As mentioned above, the aim of coordination activities is also to limit overlaps and duplication of work. In this regard, the external survey with non-EU stakeholders indicated a positive perception overall.

According to most respondents, there are overlaps only to a small or moderate extent, or not at all, with 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

all organisations included in Figure 22. There seems to be more risk of overlaps with the UN and the CoE, compared to the other organisations.

Figure 22: Overlaps with other organisations (external non-EU survey, N=76)

According to one survey respondent, the risk of overlaps is limited as the mandates differ for many of the organisations concerned. Cooperation with other institutions has become more centralised through the senior policy managers appointed in 2016, working closely with the Heads of Departments in the three departments concerned. Sometimes one team/department may not be aware of the cooperation and communication with the same stakeholder and another FRA team. To address this, internal coordination could be improved (it is likely to happen with the senior policy manager). Furthermore, having specific staff for liaising with stakeholders was reported to have had a positive impact on cooperation and coordination.

Cooperation has improved over the years, according to many survey respondents. Several respondents suggested that this is due to growing international recognition of FRA’s work. There has been an increase in cooperation with relevant UN bodies, CoE and EU agencies in the area of migration to ensure coordination of the Agency’s increasing work in this field. Regarding Member States, coope ration with NHRIs and equality bodies takes place on a regular basis, while cooperation with NCAs and national statistical offices only happens on an ad-hoc basis. Equinet was mentioned as a positive cooperation and good network at the EU and national level. The level of cooperation also depends on the stakeholders. For example, cooperation has not increased with all Member States as they may have different views on

FRA’s work.

Communication and presence in the Member States were mentioned as aspects which FRA could improve. According to one interviewee, the Agency could be more vocal and aggressive in its communication through press conferences and reports. It was also said that there could be more

continuity in the communication work. Indeed, in May–June 2017, FRA organised a high-level expert meeting on how to communicate common values, fundamental rights and freedoms. The meeting

recognised that there is an urgent need to communicate effectively in this field, and that more efforts must be made in regard to how to communicate and reach out to a larger public. Resulting recommendations included better understanding the audience, engaging with the public at all levels

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

while emphasising that human rights are for everyone, and further collaborating with the media, as well

as with literary, visual and performing arts.

To conclude, the overall message based on the data collection is that there is effective coordination and coherence. While a few overlaps have been highlighted, it seems as if complementarity between FRA and other actors is widespread. Furthermore, efforts are currently being made to improve cooperation, communication and stakeholder engagement with the aim of making the cooperation structures in place

more effective.

4.4.4. Common Approach

To what extent are the FRA’s objectives and activities coherent with the Common Approach of the European Parliament, the Council and the European Commission and the European Commission’s Roadmap for raising the effectiveness and improving governance of the

decentralised agencies

Is the Agency reporting in accordance with Commission guidance and templates?

As discussed in section 3.3.1, the Roadmap listed a number of initiatives that needed to be delivered by EU Agencies and Institutions. These initiatives were a direct translation of the Common Approach requirements, with set deadlines for implementation. The Roadmap suggested, inter alia, that the

Agencies shall:

 reinforce the link between financial and human resources and each specific action to be carried out

and make it systematic

 develop and use key performance indicators to assess their activity

 include these indicators in their individual work programmes

 develop systematic ex-post and ex-ante evaluation mechanisms.

In parallel to the actions listed in the Roadmap, the implementation of the Common Approach required the adaptation of the founding acts of existing agencies following a case-by-case analysis, as well as the

adaptation or adoption of other legislative acts (Staff Regulations 198 , Framework Financial Regulation 199 ).

By 2014, the Commission had published a series guiding document in order to assist decentralised agencies meet the requirements set in the Common Approach. These include, inter alia, the

200

Communication Handbook for the EU decentralised agencies , guidelines on the prevention and

management of conflicts of interest in EU decentralised agencies 201 and guidelines on performance

budgeting and decentralised agencies 202 . Finally, the European Commission published two progress

198 OJ L 287, 29.10.2013, p. 15.

199 OJ L 328, 7.12.2013, p. 42.

200 Communication Handbook for the EU Agencies, December 2013, European Union, Europa.eu

201 Guidelines on the prevention and management of conflicts of interest in EU decentralised agencies, December 2013, European Union,

Europa.eu

202 Performance budgeting and decentralized agencies, Guidelines, December 2013, European Union, Europa.eu

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

reports on the implementation of the Common Approach on EU decentralised agencies which focused

on the progress made in the restructuring of the Agencies. 203,204

The Roadmap on the follow-up to the Common Approach on EU decentralised agencies includes 90 actions to be performed under five titles:

 I. Setting up of agencies (if new agencies are created or when existing founding acts are revised)

 II. Operation of agencies

 III. Evaluations, audits and European Anti-Fraud Office (OLAF)

 IV. Management of financial and human resources and budgetary process

 V. Human resource management

However, only 59 of these actions are to be undertaken by the Agency. For instance, the Commission or Member States are responsible for all actions under Title I ‘Setting up of agencies’, with the exception of action 8 ‘sign a headquarter agreement in accordance with the legal order of the relevant Member

State’.

In January 2015, the FRA provided an update on the completeness status of having undertaken all actions of the Roadmap. Out of the 59 actions, it reported that:

 46 actions had been completed;

 Nine actions were in progress:

o Action 13 – Review of existing Service-Level Agreements;

o Action 19 – If the Agency’s mission requires cooperation with authorities of third countries, adopt an international relations strategy, in principle embedded in the annual and / or multiannual work programme;

o Action 20 – Submit specific initiatives with an international dimension (e.g. administrative

arrangements with third countries) to the approval of the Management Board;

o Action 36 – Adopt appropriate decisions on the creation or handling of EU classified information;

o Action 62 – Publicise the fraud prevention measures taken, and in particular, make

information on OLAF’s role and on the Fraud Notification System easily available on the Intraand

Internet sites;

o Action 67 – Inform newly recruited staff on OLAF’s role;

o Action 74 – Exchange best practices (on Activity-Based Budgeting);

o Action 75 – Pursue the development of an ABB/ABM toolbox;

o Action 76 – Develop guidelines / a training to support agencies to better apply ABB/ABM, covering key performance indicators to be included in work programmes

203 Commission progress report on the implementation of the Common Approach, December 2013, European Union, Europa.eu

204 Report From The Commission, Progress report on the implementation of the Common Approach on EU decentralised agencies, European

Commission, Brussels, 24.4.2015, COM(2015) 179 final i 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

 One action had not been started (action 71 – Better exploit the possibility offered by the

Framework Financial Regulation to ‘use joint procurement procedures with contracting authorities of the host Member State to cover its administrative needs’), and

 Three actions were not applicable:

o Action 17 – Sign working arrangements between agencies and partner DGs to ensure that they operate within their mandate and the institutional framework in the context of international activities;

o Action 18 – Provide mutual early information on international activities;

o Action 85 – Submit to the EP, Council and the Commission, an annual report on the execution of their budget and consider recommendations.

At that stage, the Agency was overall very much in line not only with the Common Approach but also with other standards from the Internal Audit Service (IAS). The activity-based budgeting (ABB) and activity-based costing (ABC) for instance were developed in 2013 by the FRA, in collaboration with the IAS and in line with the Common Approach. This was an effort towards a wider Activity-Based Management (ABM) process, whereby indirect costs (e.g. salaries, rent, electricity, telecommunications) are allocated to the operational areas of activities. This format provides better understanding of the Agency’s activities and make the analysis and interpretation easier as it links finances and thematic areas together. Furthermore, the Agency received confirmation in 2015, 2014 and 2013 that its annual accounts fairly presented its financial positions and the results of its operations and its cash flows, in accordance with the provisions of its Financial Regulation and the accounting rules adopted by the Commission’s accounting officer. On examination, the observations of the European Court of Auditors were deemed not significant. Similarly, in 2015, 2014 and 2013 the European Court of Auditors considered that the audit evidence obtained was sufficient and appropriate to provide a basis for its

statement of assurance.

Since the 2015 update, a number of changes have taken place. Most recently, the Agency has addressed the one action for which no action had been taken. In April 2017, the FRA joined seven other agencies under the umbrella of Eurofound to publish a joint call for tenders for the provision of evaluation and

feedback services (action 71). 205

Of the actions in progress, five actions have been completed:

 Action 20 – Submit specific initiatives with an international dimension (e.g. administrative

arrangements with third countries) to the approval of the Management Board;

 Action 36 – Adopt appropriate decisions on the creation or handling of EU classified information;

 Action 74 – Exchange best practices (on Activity-Based Budgeting);

 Action 75 – Pursue the development of an ABB/ABM toolbox;

 Action 76 – Develop guidelines / a training to support agencies to better apply ABB/ABM, covering

key performance indicators to be included in work programmes.

205 European Foundation for the Improvement of Living and Working Conditions – 171910/4290 Procurement documents - title of the contract:

Provision of evaluation and feedback services 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Overall, 52 actions have therefore been completed, with a further six in progress. Finally, three actions are not applicable to the Agency.

As discussed in section 3.3.1, four actions relating to the governance of the Agency cannot be fully addressed without a change to the Founding Regulation. These are:

 Action 29 – Draw up multi-annual strategic programmes or guidelines linked with multi-annual

resource planning (budget and staff in particular) and linked with successive annual work programmes.

 Action 38 – Maintain relations with national agencies and define clearly the respective roles.

 Action 89 – Develop guidelines on tailored performance indicators to assess the results achieved by

Directors.

 Action 90 – Appoint members of the boards in light of their knowledge of the Agency’s core

business and taking into account relevant managerial, administrative and budgetary skills and limit their turnover.

In 2017, the EU Agencies Network (EUAN) developed an evidence-based report reviewing the effects of the introduction of the Common Approach. 206 The report found that, while the introduction of the Common Approach had a positive impact in terms of accountability and transparency, its implementation had a negative impact on the efficiency of the agencies. This is particularity related to the increased planning and reporting obligations, which are not always aligned with the needs and work practices of the agencies. As an example, action 90, relating to the membership of the management board, cannot be seen to apply to the FRA, given the need for members of the board to be independent. The independence of the FRA also raises the question of agency’s evaluation and who should commission the regular evaluations. According to the Founding Regulation, ‘the Agency should fulfil its

tasks in complete independence’. 207

The independence of the Agency also creates questions as to whether all actions in the Roadmap can be applicable to the Agency. Beyond the question of the Management Board mentioned above, the role of who should commission the Agency’s evaluation should also be questioned. According to the Founding Regulation, and inherent to its independence, the Terms of Reference of the Agency’s independent

external evaluation should be issued by the Management Board ‘in agreement with’ the Commission. 208

Performance Development Network

The Performance Development Network (PDN) was formally established in 2011 as a ‘sub-network’. Its parent network, ‘Network of EU Agencies’, was established by the directors of the Agencies in the face of the developments that were introduced by the Inter-Institutional Working Group (IIWG). The operation and proceedings of the Network of EU Agencies are internal to the agencies and therefore do not have a publicly available presence. The resources for the administrative support are provided by the agencies. The Network of EU Agencies is coordinated by a rolling ‘troika’ of three agencies that rotate every year. A number of sub-networks exists under the overall network which focus on various subjects across agencies: Administration and Finance; Information and Communication; Legal support;

Information Technology; HR; and Performance Development Network (PDN).

206 EU Agencies network, Review of the Common Approach, Final report, January 2017.

207 Article 16 of the Founding Regulation.

208 Article 30 of the Founding Regulation.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

The Performance Development Network serves as a forum of exchange of good practice and development of common tools, guidelines and standards. It follows an annual work programme of agreed activities taking into account the overall needs of the EU agencies. The annual work programme is formally approved by the Network of EU Agencies. After the publication of the Common Approach and Roadmap in 2012, the Performance Development Network developed templates to implement

‘Roadmap’ obligations as well as requested tools and guidelines.

The PDN assisted the harmonisation of the Agencies by providing, inter alia:

 performance indicators;

 an Activity-Based Management (ABB, ABC, ABM) framework;

 guidelines for the elaboration of Single Programming Documents and Consolidated Annual Activity

Report;

 a common glossary and benchmarking information (included in the Evaluation handbook for

Agencies);

 a suggested format and process of evaluations, including stakeholder consultation guidelines;

 a quality management system.

The FRA developed a Performance Measurement Framework in 2011 following an Internal Audit Service on Planning and Monitoring, and further revised it in 2013 and 2015 to support its monitoring and evaluation activities by assessing its performance and by demonstrating its achievements, results and impact. The development of the Performance Measurement Framework is a result of the recommendations from FRA’s first external evaluation and the guidance produced by EU Institutions under the Common Approach for decentralised agencies described above. According to FRA, ‘the PMF brings all performance-related information and data under a logical framework, representing the reference point for monitoring and reporting on FRA’s performance, results and achievements, and is underpinned by a logic of intervention’. It includes a logic model (composed of outputs, immediate,

209

intermediate and ultimate outcomes) and the list of the performance indicators.

209 Annual Work Programme 2016, FRA, December 2015

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Efficiency

Efficiency looks at the relationship between resources used by an intervention and the changes generated by the intervention

(which can be either positive or negative).

The concept of efficiency is also tied to cost-effectiveness, which refers to the relationship between the inputs for or costs incurred by an intervention, a programme of activities or an organisation and the outcomes which are achieved as a result.

The money which the Agency receives is used to acquire inputs (financial and human resources, etc.), which are used to provide

a range of activities and outputs. The Agency’s budget is broken down into staffing costs (the vast majority of this being expenditure on ‘staff in active employment’), administrative expenditure (e.g. rental, consumables) and operational expenditure

(operational coordination and capacity building). Tied into the expenditures is the cost of producing the Agency’s outputs (i .e. research report, an Opinion report, fulfilling a data request, etc.).

Assessing the efficiency of individual activities is dependent on the availability of data on those activities, such as the number of person hours required to produce an output (i.e. one research report). There also need to be clear and logical workflows that are documented as part of a logic model. Activities and outputs that are produced by the Agency vary by department and by thematic area, and as such each of the four FRA departments must be assessed for their efficiency, as well as the Agency as a whole. As well as assessing the internal efficiency of the Agency, it is also important to assess the external efficiency. In order to understand the efficiency of the organisation in relation to its use of financial and human resources and the cost of its outputs

and activities, it is necessary to undertake an analysis of both the Agency’s internal and external efficiency. Assessing the

efficiency of internal processes will use the cost of outputs to identify the financial benefits associated with synergies with the Agency’s stakeholders, in particular the end users (i.e. Member States). The aim will be to identify the net benefits associa ted with the Agency’s activities when compared to an alternative environment where similar act ivities are undertaken at an individual Member State level or not at all. This analysis will also highlight the net benefits of using the evidence base provided by FRA for decision-making versus the potential costs associated with the risk to wider society due to (potentially poorer) decision-making occurring in the absence of high-quality evidence. With the information provided by this analysis, the Agency will be able to consider the effects and financial implications of modifying its tasks.

It is also important to examine the Agency’s alignment with key actions and strategies aimed at improving efficiency, in particular those aimed at improving the efficiency of EU decentralised agencies.

4.5.1. Does the Agency’s organisational and budgetary structure contribute to

effectiveness and efficiency of its operations?

The question to be answered here is the impact of the Agency’s organisational structure, and in

particular the recent changes to its structure, on its efficiency, linked to the effectiveness of its actions.

The assessment has been undertaken through the use of opinions collected in the both the internal and

external stakeholder’s surveys, interviews with FRA staff and external stakeholders, as well as through

the assessment of relevant documents.

During the 2013–2017 period the Agency has undertaken three changes to the organisational structure.

In 2014, the change to the organisation’s structure related to the adoption of new policy in regard to

engagement and use of temporary staff. In December 2016, the Agency’s structure was simplified to reduce the number of departments, combining the administration and the human resources and planning departments to create the corporate services department, whilst also introducing the roles of Senior Policy managers in the three operational departments. Furthermore, the Communication and outreach Department was also reorganised and renamed the Fundamental Rights Promotion

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Department (FRDP) in 2016. A full description of the Agency’s organisation and the changes undertaken can be found in section 2.3.

When responding to the internal survey question whether the Agency’s organisational structural changes that took place in 2016/2017 had an impact internally on the Agency’s operations, 18 of the 36 internal stakeholders believed these structural changes had a positive impact on the Agency’s internal operations. However, 12 of these respondents qualified that the impact of the change was limited, that any impact would take more time to materialise, given the short time period that has elapsed since these changes, that the impacts were not what was intended or that significant challenges are still not addressed. Nine of the internal stakeholders believed that the changes had no impact internally on the Agency’s operations whilst seven stakeholders believed it was too early to assess any impact at this stage. The changes to the Agency’s structure were communicated and followed a consultation with the relevant Departments. Despite this, a limited number of respondents discussed the implementation of these structural changes, reporting disruption to work flows in part due to a lack of clarity over new responsibilities, and they expressed a desire to see better management of transition periods. When investigating the impact of the creation of the Corporate Services Department and that of the Fundamental Rights Promotion Department individually, several survey respondents discussed a different level of impact. They noted that, since there were few changes in the FRPD, there has been a limited impact on the Agency’s internal operations at this stage. These respondents also noted the ‘clearly positive changes’ to the Agency’s internal operation brought about by the creation of the

Corporate Services Department.

Respondents noted that the merger of human resources and planning with administration was necessary. Several respondents raised concern over limited communication to staff on the changes and the new lines of responsibility that these structural changes entailed, highlighting frustration amongst a

number of staff over the communication of these changes.

In terms of positive impacts of the change, respondents noted that the reorganisation strengthened the Agency’s engagement on fundamental rights promotion and awareness raising, streamlined pending tasks such as promotions and recruitments and brought to the Agency integrated management of the support processes. Additionally, the creation of self-standing roles for communication and stakeholder engagement activities was considered ‘very helpful’ in contributing to the effectiveness and efficiency of

its operations.

In terms of capacity building, the creation of the FRPD raised questions over the role of the Department’s staff in supporting the research projects of the Agency with their expertise, in terms of improving the dissemination of this research at the research design stage implementing a more targeted approach for FRA outputs, and the feasibility of them continuing this role alongside their new

responsibilities as part of the FRPD.

In terms of potential negative impacts, whilst just a small number of respondents categorised the changes as such, these respondents focused on the concerns raised by the majority of survey respondents that regarded the changes as having a positive impact. These respondents similarly highlighted that the changes have created ‘confusion concerning responsibilities, more workload for research and disruption in activities’ and whilst not discounting future potential benefits, stated that

these have not begun to materialise yet.

The evaluation findings point towards a positive long-term impact on the efficiency of the Agency’s operations of changes to the Agency’s organisational structure, new roles and a renewed commitment to improving the Agency’s outreach. However, whilst staff acknowledge the important step the changes

represent, significant disruption and lack of clarity still exist. In the short term, the Agency should invest

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

resources to complete the integration of these departments and roles, clearly delineating any changes in responsibilities to staff and the impact on their own internal processes, in order to fully capitalise on the benefits these changes have produced.

4.5.2. Is the size of the budget and human resources appropriate and proportional to what FRA is expected to achieve? Is it sufficient for reaching a critical mass of impact?

The question of whether the Agency’s budget and resources are appropriate and proportionate to what the Agency is expected to achieve depends to a large extent on what is understood by impact as well as the understanding of the Agency’s role.

Interviewees were divided, particularly amongst internal stakeholders and EU institutional stakeholders in one group against other external stakeholders in another, over whether the size of the Agency’s budget was appropriate and proportional to what the Agency is expected to achieve. Most internal stakeholders interviewed felt the Agency’s budget was not adequate in order to fulfil its multiple roles, with a number of interviewees regarding some of the Agency’s objectives as unattainable at this level of resourcing.

Regarding the Agency’s task to ‘develop a communication strategy and promote dialogue with civil society, in order to raise public awareness of fundamental rights and actively disseminate information about its work’ and its objective to ‘provide the relevant institutions, bodies, offices and agencies of the Community and its

Member States’, 210 stakeholders believed the Agency requires ‘ The FRA has to make choices, they have a limited budget. Some want FRA to do

further human resources and an increased budget, in order to everything, but it is simply not possible .’

achieve these. In particular, internal survey respondents

mentioned that the FRPD needed significantly more resources to Member of the LIBE committee

fulfil their role, especially in terms of creating a multiplier effect at the national level through having staff to monitor and disseminate its findings across and throughout the 28 Member States. This was supported by interviews with civil society members, including Advisory Panel Members, who described an inadequate allocation of resources to engage with civil society at the national level and a need for an increase in capacity to sufficiently meet the needs of this relevant stakeholder group. Similarly, Management Board members highlighted that the Agency needs significantly more resources for its communication activities as well as more resources to fulfil their mandate regarding the promotion of dialogue with civil society. However, more respondents to the internal FRA survey indicated that resources for research were only met to a ‘moderate extent’ or to a ‘small extent’ in comparison with other areas of the Agency’s work, while for communication activities and corporate services many internal respondents indicated that they ‘did not know’ if resources were sufficient.

One Fundamental Rights Platform member raised the concern that the Agency’s limited resources affect the research methodology and quality of the Agency’s research activities. Whilst it is possible to undertake some tasks with a reduced budget, for example using videoconferencing rather than travelling, others that require a staff member to visit a country or a survey to be conducted, mean that any reduction in resources affects the ability of the Agency to fulfil its mandate in this regard.

210 Council Regulation (EC) No 168/2007 i of 15 February 2007 establishing a European Union Agency for Fundamental Rights

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Regarding the Agency’s task to ‘formulate and publish conclusions and opinions on specific thematic topics, for the Union institutions and the Member States’, internal stakeholders expressed a belief that the Agency’s budget and human resources were not proportionate to the task. Management Board members discussed the fact that the Agency’s activities and requests for opinions from the EU Institutions had increased, leaving the Agency with an increased workload and a static budget that, unless resourcing is increased, will lead to the Agency’s resources being insufficient to fulfil its mandate. This was supported in the internal survey, where it was highlighted that unlike ‘other JHA agencies which are getting extra budget for responding to new needs (e.g. refugee crisis), FRA is relying on the same annual budget despite more tasks’. This is highlighted by the fact that the Agency saw a six-fold increase in official requests by EU Institutions for legal opinions to be drafted by the Agency in the field of asylum, migration and border control in 2016. Similarly, the Agency organised 124 events in 2016, more than double the 60 events it organised in 2015. Furthermore, FRA offered expertise at 484 presentations and hearings at the EU and Member State level in 2016, again more than double the 240 presentations and hearings the Agency attended in 2015.

Many civil society members felt unable to offer an informed answer to the question of budget but were of the opinion that the Agency had an appropriate budget for its tasks based on discussions with Agency staff they collaborated with. Several international organisations and civil society members felt that the Agency’s resources were

proportionate for what the Agency is expected to achieve, but it ‘ The main challenge last year was the issue of priority setting .’

was the prioritisation of these existing resources which the

Agency had difficulty with, which highlighted a lack of FRA staff member

understanding of the prioritisation process discussed by several interviewees from NHRIs, international organisations and civil society members.

However, when broken down into the areas of activity of research, communication activities and corporate services, the majority of respondents to the survey believed that the budget for research activities is sufficient only to a moderate or small extent. An important proportion of respondents state that they ‘do not know’ if the budget for communication activities or corporate services are sufficient to achieve FRA’s objectives in the past five years.

Figure 23: Internal survey responses to question ‘To what extent has the FRA received sufficient budget to achieve its objectives in the past 5 years in the following areas of activity?’ (N=66)

Communication activities

Corporate services

Research activities

0 10 20 30 40 50 60 70

To a large extent To a moderate extent To a small extent

Not at all Do not know No Answer 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

4.5.3. Is there a good balance between administrative and operational budget?

The assessment of the balance between the Agency’s administrative and operational budget has been undertaken through the use of opinions collected in both the internal and external stakeholders’ surveys, interviews with FRA staff and external stakeholders as well as through a comparison with other selected EU decentralised agencies. The benchmarking was undertaken with three separate EU agencies: EASO, EMCDDA and Eurojust. EASO and EMCDDA were chosen due to their similar sizes to FRA as well as to similarities in the subject area and tasks undertaken by the agencies in research and coordination. Eurojust was chosen due to its role in judicial and police cooperation in criminal matters as well as its similarity in size to FRA. Similarly, EASO was included to highlight the increasingly significant difference in scale between the agencies despite their overlapping area of focus and their prominent

roles in the EU’s response to the so-called migrant crisis in Italy and Greece.

Table 19: Comparison between FRA 211 , EASO 212 , EMCDDA 213 and Eurojust 214 total, staff, operational and operating

budget (EUR)

2013 2014 2015 2016 2017 Total Budget

FRA 21,348,510 21,229,000 21,229,000 21,359,000 22,704,500 EASO 12,000,000 14,656,000 14,991,360 19,438,600 69,206,000 EIGE 7,696,516 7,716,103 7,658,166 7,628,000 7,628,000 EMCDDA 16,057,482 15,183,962 15,333,962 15,393,962 15,807,164 Eurojust 32,358,660 32,449,671 32,994,000 43,539,737 47,879,237

Staff Budget FRA 11,513,000 12,185,000 11,961,000 11,799,000 13,069,500 EASO 5,044,000 6,143,000 6,176,360 9,126,500 16,521,014 EIGE N/A N/A 3,034,551 3,270,765 3,190,000 EMCDDA 9,537,228 9,336,638 9,196,364 9,321,198 10,128,023 Eurojust 17,333,166 16,964,992 16,976,924 18,864,895 18,501,261

Operational Budget FRA 7,628,510 6,800,000 7,050,000 7,328,000 7,499,000 EASO 5,000,000 6,027,000 6,178,000 6,609,000 43,804,986 EIGE N/A N/A 1,065,480 1,056,168 996,000 EMCDDA 4,322,017 3,777,495 3,887,623 4,348,475 4,452,593 Eurojust 7,751,294 8,056,979 8,399,464 8,196,156 8,276,674

Support/Operating Budget FRA 2,207,000 1,096,000 2,218,000 2,232,000 2,136,000 EASO 1,956,000 2,486,000 2,637,000 3,703,100 8,879,999 EIGE N/A N/A 3,695,893 3,357,266 3,442,000 EMCDDA 2,198,238 2,069,828 2,249,974 1,724,288 1,226,547 Eurojust 7,274,200 7,427,700 7,617,612 16,478,686 21,601,302

It is very difficult to meaningfully compare the resources used by the FRA and other Agencies given their

differences. The FRA’s operational budget mainly relates to staff salary (calculated using Activity-Based

211 All annual budgets available at: http://fra.europa.eu/en/about-fra/finance-and-budget/financial-documents

212 All annual budgets available at: https://www.easo.europa.eu/budget-finance-and-accounting

213 All annual budgets available at: http://www.emcdda.europa.eu/publications-seriestype/budget

214 All annual budgets available at: http://www.eurojust.europa.eu/doclibrary/budget-finance/Pages/annual-budgets.aspx

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Management) while the operational budget of EASO or Frontex will include activities such as chartering ships. As such, whilst the operational expenditure of the Agency is significantly lower than that of EASO in both nominal and percentage terms (just over €7 million, or 33%, for FRA in 2017 against over €43 million, or 63% for EASO), in comparison to the other selected EU agencies it is not significantly lower, falling into a similar range in nominal and percentage terms as EMCDDA but significantly more than Eurojust in percentage terms at 17%. However, staff expenditure as a percentage of the total Agency budget, approximately 58% in 2017, is in line with the EMCDDA, a benchmarked agency of a similar size,

at 64% in 2017.

Table 20: Overview of budgetary changes to select EU agencies

Agency FRA EASO EMCDDA Eurojust

Budget changes (%)

2013 to 2017 6.35% 576.7% (1.56%) 47.96%

Budget changes (EUR) €1,355,990 €57,206,000 (€250,318) €15,520,577

Whilst FRA’s 6.35% budget increase during the 2013–2017 programming period was greater than that of the EMCDDA’s modest budget reduction of 1.56%, it is dwarfed by the scale of budget increase to EASO, whose aim is to facilitate improved protection for asylum seekers and coordination between member states and Eurojust, which seeks to improve investigative and prosecutorial coordination among agencies of the EU Member States. In light of its additional responsibilities with regard to the so-called migrant crisis that erupted in 2015, EASO experienced an approximately 577% increase in its budget.

Similarly, Eurojust experienced an approximately 48% increase in its budget in the same period.

Regarding other operational efficiency indicators, the Agency is line with other selected agencies in terms of budgetary execution with over 99% of overall budget consumption of the EU subsidy

throughout the programming period.

Table 21: FRA consumption of EU budget 2013–2016

FRA 2013 2014 2015 2016

Level of budgetary consumption of the EU 99% 99% 99% 99% subsidy

To conclude, the Agency’s balance between operational and administrative budget is in line with comparable EU decentralised agencies. Thanks to the introduction of Activity-Based Management, the Agency has maintained a high-level budgetary consumption of its EU subsidy throughout the

programming period, indicating an efficient budgetary execution.

4.5.4. To what extent has the Agency been successful in creating synergies and an optimal use of combined resources allocated for the implementation of its mandate and tasks to manage the operation?

The majority of internal survey respondents felt that the Agency’s shift in the allocation of staff across different sectors had impacted the Agency’s internal operations.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Several respondents highlighted that this was not happening to the extent necessary, or that the impact had yet to be realised and visible. The flexibility to allow staff working in other sectors to support migration/asylum work had been very useful to deliver on the increasing demands in this area; however, staff who temporarily supported the Agency’s work on the migration crisis did this in addition to their other tasks, which several respondents discussed as not being deprioritised in turn, therefore quite

significantly increasing their workloads.

One internal stakeholder, citing the Agency’s shift in the allocation of staff, said a positive impact had been achieved but it was limited because the main challenge, workload of existing staff, has not been addressed. This was supported by another respondent who categorised the positive impact of the shift in terms of more accurately matching tasks with staff’s skills and experience against the fact that colleagues were now spread even more thinly. Similarly, another respondent cited a need for a new human resources capacity assessment to assign/train or even replace staff to perform the needed tasks

across the Agency in line with these different responsibilities.

Several internal survey respondents discussed the synergies and efficiency gains enabled by the merger of the Corporate Services Department that would be create a potential redeployment of staff to

operational departments.

The Agency has been successful in creating synergies through cooperation and coordination with organisations in the fundamental rights field, but considerable opportunity still exists. In 2013, the Agency focused on developing synergies in the areas of communication, cooperation and of topics for FRA flagship events, on cooperation with inter-governmental organisations such as the Council of Europe, and participation in the EU’s JHA network. In 2014, the Agency documented the drive to create synergies through joining the European Commission Roma integrations indicators work party. In 2015, the Agency coordinated the JHA network, joined the working party on Improving Reporting and Recording of Hate Crime in the EU and chaired the Roma integrations indicators work party throughout the year. In 2015, for the first time the Agency hosted its Network of National Stakeholders and FRA bodies in Vienna in November to identify more synergies and opportunities for cooperation. In 2016, it notably participated in an operational meeting between the ODIHR and the FRA Management to identify synergies in the work programmes and areas of cooperation. Similarly, the Agency’s multiannual cooperation with the Norway Grants organisation is a strong example of synergies gained through the successful implementation of cooperation agreements and coordination to maximise the efficiency of FRA activities, outputs and events, including through the sponsoring of attendance of FRA events and financing an Agency event for the EU Presidency. Regarding the creation of resource synergies, the Agency’s joint procurement tendering for the provision of evaluation and feedback services alongside seven other EU Agencies is further evidence of the Agency’s attempts to create

synergies alongside existing EU Institutions in order to optimise its limited resources.

Interviewees from inter-governmental organisations and civil society organisations stressed that cooperation with the Agency has created substantial synergies and efficiency benefits, which has benefited both organisations and their respective stakeholders, including the joint production and dissemination of outputs such as handbooks and events as well as conducting national-level research. Civil society interviewees report a desire to improve the efficiency of the Agency’s existing resources, in the form of the raw comparable data the FRA has collected, to undertake further analysis in the fundamental rights field with and in a role for them in multiplying the FRA’s communication and dissemination activities at the national level, indicating that a significant opportunity for further synergies exists. Respondents from inter-governmental organisations similarly reported the substantial

opportunity for further synergies that existed between the Agency and the organisations in further

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

coordination between events and outputs, and in particular in communication and dissemination activities.

Overall, the Agency has been successful in ensuring the optimal use of combined resources for the implementation of its mandate and tasks to manage the operation. However, considerable opportunity

exists for further synergies with civil society organisations and other inter-governmental organisations.

4.5.5. To what extent do internal processes and ways of working impact on the Agency’s ability to perform its essential activities?

In order to assess this evaluation question, the internal stakeholder survey, which includes staff, Management Board Members and Scientific Committee members, was examined alongside the extensive interview programme with FRANET and other organisations that had an overview of the

Agency’s internal processes.

Regarding the effectiveness of the Agency’s internal communication systems and processes in supporting staff to meet the Agency’s objectives, the majority of internal stakeholders believed that

they were effective to a large extent or to a moderate extent.

Figure 23: Internal survey question ‘To what extent are the Agency’s internal communications systems and processes effective in supporting staff to meet the Agency’s objectives?’ (n=66)

6.06% 1.52%

3.03%

24.24%

31.82%

33.33%

To a large extent To a moderate extent To a small extent

Not at all Do not know No Answer

A substantial majority of internal stakeholders believe that the quality control mechanisms in place at the Agency are to a large or moderate extent effective in maintain the high scientific quality of the Agency’s work. Internal stakeholder interviewees discussed a rigorous quality control system with documentation for staff that covered approximately 90% of the processes, a strong internal control committee, FRACO, the Agency’s control chain and the fact that outputs are cross-checked by heads of department. Internal stakeholders discussed an efficient Scientific Committee that was successful in

maintaining a minimum quality standard at the Agency.

However, several academic and civil society organisations discussed the fact that the intended audience for the Agency’s outputs is not always clear as these outputs typically do not follow the same structure

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

and logic as academic articles. However, the raw data from FRA’s surveys is placed in the public domain for researchers to re-use for their own research purposes, which is in contrast to the restricted access to datasets typically found in academic articles. Additionally, FRA research staff have used research

undertaken by the Agency and the data that it provides in order to publish in academic journals.

Several civil society stakeholders, NLOs and NHRI members discussed instances of discovering incorrect national-level information in FRA reports (not survey findings) that they believed was due, in part, to the Agency’s selection of FRANET research partner in that Member State and questioned how the Agency ensures the accuracy of data collected by all its FRANET members. Similarly, internal stakeholders expressed concern about a drive for efficiency that was making the balance between maintaining quality and the quantity of outputs increasingly difficult, indicating a considerable strain on resources. However, it is worth noting that these external interviewees were unaware of the quality control mechanisms

used by the Agency with regard to its output.

Figure 24: Internal survey question ‘To what extent are the quality control mechanisms in place effective in ensuring high scientific quality of the work done and outputs produced by the Agency?’ (n=66)

10.61% 1.52%

3.03%

13.64% 51.52%

21.21%

To a large extent To a moderate extent To a small extent

Not at all Do not know No Answer

A majority of internal stakeholders believed that, to a large or moderate extent, the monitoring, reporting and evaluating of the Agency were adequate for an appropriate assessment of performance in

the context of the Agency system .

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 25: Internal survey question ‘To what extent are the mechanisms for monitoring, reporting and evaluating the Agency adequate for an appropriate assessment of performance in the context of the Agency system?’ (n=66)

4.55%

18.18% 30.30%

4.55%

12.12%

31.82%

To a large extent To a moderate extent To a small extent

Not at all Do not know No Answer

A substantial majority of internal stakeholders believe that, to a large or a moderate extent, FRA staff are open to change and responsive to new ways of working. Internal stakeholders interviewed were more divided over FRA staff’s openness to change but, similarly, the majority expressed a desire to see

improvement at the Agency.

Figure 26: Internal survey question ‘In your opinion, to what extent are FRA staff open to change and responsive to new ways of working?’ (n=66)

4.55% 6.06%

4.55%

27.27%

10.61%

46.97%

To a large extent To a moderate extent To a small extent

Not at all Do not know No Answer

Governance of the Agency

As described above, the Agency’s governance structure comprises a Management Board assisted by an Executive Board. The Management Board is in turn responsible for the appointment of the Director.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

In order to ensure the Agency’s independence, the members of the Management Board (one per Member State, one appointed by the Council of Europe and two representatives of the European Commission) are appointed for a non-renewable mandate of five years. While overall the Management Board operates in an efficient manner, there are a few areas which could be clarified in order to ensure the Agency follows a closely as possible the principles of efficiency set out in the Common Approach.

These are as follows:

 The Regulation is unclear as to whether it is possible for a member to serve two non-consecutive

terms in the Management Board.

 Member States have different ways of selecting their member of the Management Board. The point

in time at which their mandate should be calculated is unclear (i.e. whether their terms starts as soon as their predecessor’s terms expire or when they are finally appointed).

The position of Director is a central one to the Agency. Unlike for a number of European Decentralised agencies, the FRA’s Director is appointed externally for a period of five years extendable by three years on the basis of ‘his or her personal merit, experience in the field of fundamental rights and

administrative and management skills’. 215 In order to ensure the independence of the Agency, the

procedure for the selection of the Director is a lengthy process that can last over a year. First, the Commission draws a list of candidates after the publication of a call for candidates. The candidates are then asked to address the Council as well as the Parliament, who then provide their opinion by stating their order of preference. The final decision is then made by the Management Board, taking into account the opinions of the Council and the Parliament. In practice, the previous Director, Morten Kjaerum left the Agency on 31 March 2015 and was replaced by interim Director Constantinos Manolopoulos until the appointment of Michael O’Flaherty starting on 16 December 2015. Given that the vacancy notice was published in the Official Journal of the European Union in October 2014, the

length of the procedure is deemed by an important number of stakeholders as too long.

Notwithstanding the interviewees’ overwhelmingly positive view of both the interim and the current Directors, a number of issues relating to the selection process were raised:

 However effective the interim Director can be, the role is not political, and key political decisions

could not be taken during the interim period. If the selection process cannot be shortened, it should be initiated earlier to ensure that the interim period is as short as possible.

 The Director’s maximum term of eight years does not entitle her or him to the European Union’s full

pension. Given that high-level roles for suitable candidates with the adequate knowledge of Fundamental Rights and the necessary management experience are few and far between, it would be very difficult for a Director serving the additional three years to turn down an opportunity.

Similarly, it has the potential to reduce the pool of potential candidates.

In conclusion, internal stakeholders are satisfied with the impact on the Agency’s ability to perform its essential activities of internal processes and ways of working, believing that the Agency has a rigorous quality control system, strong internal communication channels and effective mechanisms for monitoring, reporting and evaluating the Agency. Additionally, internal stakeholders are open to change and responsive to new ways of working – especially important given the recent restructuring of the

Agency and the creation of new responsibilities within existing roles.

215 Founding Regulation, Article 15(1).

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

4.5.6. To what extent have the effects of the Agency’s activities been achieved at lower costs because of the Agency’s intervention?

When examining this evaluation question, interviews with external stakeholders, including those from EU Institutions and bodies, were included. When assessing the extent to which activities have been achieved at a lower cost due to the Agency’s intervention it is important to assess what can be measured, as well as the more intangible benefits of the Agency and what would have occurred if it did

not exist: the counter-factual.

The Agency has been able to gather comparative research at the national level in all Member States at a significantly reduced cost as compared to civil society or national authorities, for which it would be prohibitively expensive to produce such sets of comparable data. As described in a recent European

Parliament study 216 on the cost of non-agencies, research suggests that it is ‘considerably less costly to

carry out the tasks assigned to the agencies at the EU level than it would be if these tasks were undertaken by the EU28 Member States’. The tangible benefit is the difference in cost between the sum of each national authority collecting their national-level data as well as combining it, and the cost of the EU-level survey for the Agency. There are considerable cost savings in the Agency’s elimination of duplication of analysis and research infrastructure. Additionally, FRANET, the Agency’s multi-disciplinary research network, uses contractors to conduct research at the national level, significantly reducing the cost of building and maintaining the research capacity in all Member States, and therefore the EU and national-level stakeholders benefit from the lower cost of production of this research as opposed to other existing institutions or the use of in-house research capacity instead of contractors. Regarding intangible benefits, the comparative set of data across multiple thematic areas that the Agency has created did not exist before the Agency came into being, and therefore the counter-factual is the cost of

not having this data to EU citizens.

Another intangible benefit is the Agency’s provision of services to EU Institutions and its role in relation to international organisations as well. The Agency’s cooperation with inter-governmental organisations in the field of fundamental rights, including ODIHR, COE and UNHCR, acts as the main dialogue between the EU Institutions, through the European Union’s dedicated Agency for Fundamental Rights, with these important international-level stakeholders, across several thematic areas. The cost of no Agency would be the loss of the Agency’s past, current and future contribution to developing standards and indicator frameworks at the international level in these thematic areas. While difficult to quantify, this would be a

significant loss to the fundamental rights field.

Potentially the most difficult to quantify is the impact the Agency has had on improving the fundamental rights situation of European citizens and remit-specific end users, despite these not being the Agency’s primary target groups, as described in the Agency’s mandate. The Agency’s outputs, opinions and assistance have brought awareness of fundamental rights issues and both directly and indirectly impacted the protection, monitoring and awareness of European citizens’ fundamental rights. Both internal and external stakeholders agree the fundamental rights situation in the EU has improved due to the existence of a dedicated institution at the EU level to address these concerns. The counter-factual

cost of having no agency with a purview on fundamental rights in the EU is likely to include a

216 European Parliament, ‘The cost of Non-Agencies with Relevance to the Internal Market’, 2016. Available at:

http://www.europarl.europa.eu/RegData/etudes/STUD/2016/572702/IPOL_STU(2016)572702_EN.pdf 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

deteriorated fundamental rights situation at the EU level and any resulting cost to European citizens in the reduction of their fundamental rights.

The Agency’s substantial impact on the development of standards for European Institutions has been achieved with moderate resources. One key example has been the Agency’s work in the Greek and Italian hotspots, contributing significantly to creating standards for other EU Institutions and national authorities to properly respect fundamental rights in the execution of their tasks, particularly with regard to unaccompanied children. The Agency was able to undertake this project at a moderate expense but achieved significant results given the availability of the fundamental rights expertise at the Agency and the substantial immediate need. The other EU Institutions did not have the available expertise and, regarding the counter-factual, would have needed to incur considerable costs in the recruitment of suitable expertise and in building a research and reporting infrastructure, particularly in

terms of the monthly data reports across the hotspots.

To a significant extent the effects of the Agency’s activities have been achieved at lower costs because of the Agency’s intervention. The Agency’s role in developing standards, processes and procedures at the EU and national level, offering informal advice to the EU Institutions, its role as the guardian of fundamental rights in the EU and its impact on European citizens have all been achieved at a lower cost of intervention due to the Agency, and significant costs are associated with the counter-factual of no

fundamental rights agency at the EU level.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

• 5. 

  Conclusions and recommendations

  Conclusions

  The following conclusions are based on findings of the evaluation and are divided according to their evaluation criteria. They should be read in combination with the conclusions of the 1 st independent evaluation of the Agency.

  Relevance

  Overall, the Agency’s internal and external stakeholders believe the FRA’s objectives are relevant to the needs of duty bearers, and have been throughout the programming period. However, a substantial cross-section of the Agency’s stakeholders believe that the FRA’s mandate should be changed in order for the Agency to meet an existing fundamental rights need in the EU around police and judicial cooperation in criminal matters. The first independent external evaluation of the Agency of 2012 concluded and recommended 217 that the Founding Regulation be ‘Lisbonised’ to enable the Agency to operate in all areas of EU competence, a recommendation which was supported by the Management Board’s response on 4 June 2013 to the European Commission’s Vice -President Viviane Reding on the evaluation’s recommendations. A response from the Commission on this recommendation was not received by the Agency. However, whilst the Agency is unable to operate to the same degree and intensity in all the aspects of fundamental rights in the European Union, the situation within the Union has changed considerably during the programming period. This was supported by interviews with international, EU and national level stakeholders, and FRA staff, stakeholder consultations undertaken by the Agency, internal and external stakeholder surveys as well as substantial desk research. Similarly,

  other decentralised EU agencies, such as EIGE, have mandates that do not limit their Agency’s ability to

  undertake research in the former third pillar, allowing the Agency to operate in all policy areas, in line with the Lisbon Treaty.

  However, one key element when assessing relevance is that there is not always a common

  understanding of what the objectives of the Agency are. This lack of common understanding can lead

  to situations where some stakeholders have a more negative view of the Agency’s impact given that

  they expect its objectives to be much wider than they actually are according to its Founding Regulation.

  In terms of the thematic areas of the FRA, the majority of the duty bearers at EU and national level

  stated that the thematic areas ‘Racism, xenophobia and related intolerance’ and ‘Immigration and integration of migrants, visa and border control’ reflect the areas of their current needs. The majority of

  respondents identified the thematic area ‘police and judicial cooperation in criminal matters’ and ‘Roma integration’ (for national level respondents) and ‘Rights of the child’ for EU -level respondents, among others, as thematic areas that reflect their needs to a small extent, or do not reflect their needs at all. However, as previously addressed in this evaluation, it is impossible to interpret these results without

  taking into account, first, the available evidence on the strong impact of FRA’s work e.g. on Roma

  integration at EU level, and secondly, the composition – in particular the subject background – of the respondents to understand the level of bias given towards thematic areas that respondents themselves work in.

217 FRA, Evaluation of the European Union Agency for Fundamental Rights, Final Report, November 2012.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

In terms of the activities under the Agency’s mandate, there was a broader consensus among interviewees and survey respondents that the current activities were successfully meeting the fundamental rights needs of the EU. However, in order for the Agency to improve the extent to which it meets the fundamental rights needs of the EU through undertaking new activities, an extension of the

Agency’s mandate is required.

Effectiveness, Utility and Impact

Regarding meeting the needs of stakeholders:

FRA has been very responsive in meeting the different needs of its duty bearers. On the EU level, FRA has in 2016 provided an increasing number of opinions and responded to informal requests for information, mostly to meet the needs of the EU Institutions in the field of migration but also to react to a diversity of other issues. At the national level, FRA has increased its cooperation with national parliaments and collaboration with media and civil society was strengthened through various country

visits.

Overall stakeholders at national and EU level have found FRA’s outputs particularly useful as they reflect current needs (particularly since FRA has shifted its efforts to the migration crisis). In practice, the usefulness of FRA’s outputs is reflected in the fact that many duty bearers regularly use and reference

FRA reports.

Some of the strongest evidence as regards FRA’s EU-level impact relates to references made to its work in key policy documents, such as Council Recommendations, Commission Communications, Staff Working Documents, etc. These stem from the research and analysis undertaken at national and EU

level.

Regarding matching the objectives set in multi-annual and annual programmes:

FRA achieved all thematic and strategic objectives which were set out in the multi-annual programmes. In some areas FRA even went beyond what was necessary and engaged in additional activities to meet the needs of the current migration crisis. In regard to strategic priorities, FRA was also successful across

all six priorities:

 Enhancing FRA’s contributions to processes at EU level;

 Enhancing FRA’s contribution to processes at the national level;

 Identifying trends over time and measuring progress in EU Member States;

 Developing timely and targeted responses to fundamental rights emergencies;

 Improving the impact of FRA’s communication and awareness raising; and

 Planning FRA’s work and evaluating its impact.

Apart from this positive conclusion, providing useful inputs to stakeholders at the national level has been a challenge.

Regarding the Agency’s Communication

Generally, FRA outputs were perceived as very accessible by all interviewed stakeholders of different Member States. While there were some debates on whether FRA outputs should be translated into different languages more often, the majority of stakeholders felt that this was not necessary. One

general point on the accessibility related to what the key target group of FRA is. While some

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

stakeholders argued that FRA outputs are too technical, others argued that they are not academic enough, suggesting the Agency is unique in its output.

FRA has over the years been increasingly successful in disseminating its findings. An analysis of Facebook, Twitter and YouTube data revealed that more people followed FRA and liked relevant posts.

This is also reflected in the increasing number of downloads of FRA reports over the years.

Some stakeholders have argued that FRA should be engaging more with ‘multipliers’ in order to further increase its visibility.

The Agency can clearly demonstrate having an impact at EU and national level. These impacts stem principally from the FRA’s core body of research and analysis, upon which additional outputs and

activities can be built, such as legal opinions, country visits and missions, etc.

Added value

The FRA is contributing importantly, and in a unique way, to the promotion and protection of fundamental rights in the EU. Interviewees believed that the simultaneous EU-wide collection of data distinguished FRA from other international organisations such as the OSCE or the Council of Europe, as well as from NGOs at national level. Similarly, FRA is the only organisation collecting this type of data at European level, and the CoE relies heavily on this data. The role of FRA as a provider of relevant and unbiased data was mentioned as an added value by several interviewees. Some interviewees mentioned the Agency’s role as a watchdog, indicating that this role should be further expanded (e.g. by providing more financial resources), as no other institution has that purpose. It was also mentioned that the FRA has an important role in providing a forum for discussions on fundamental rights involving relevant stakeholders from across the EU Member States. As outlined above, the evaluators conclude that added value can be found in FRA’s research activities, the conclusions and opinions, the stakeholder engagement, the independent nature of the Agency, as well as in its (so far rather indirect) monitoring

role.

Coordination and coherence

The data collection in the context of this study indicates that there is a robust structure in place for collaboration with relevant EU/national policy stakeholders, international organisations and civil society. Furthermore, FRA has adequate informal channels in place which are used for exchange of information. The overall message based on the data collection is that there is an effective coordination and coherence, particularly with other stakeholders involved in similar activities as FRA (e.g. the Council of Europe, OSCE, etc.). While a few overlaps have been highlighted, it seems as if complementarity between FRA and other actors is widespread. Furthermore, efforts are currently being made to improve cooperation, communication and stakeholder engagement, with the aim of making the cooperation

structures in place more effective.

In general, cooperation activities with CoE and the UN are considered positive and effective according to most interviewees, and some indicated that the cooperation has improved over time. Interesting stakeholder meetings, as well as the FRA taking the initiative with other international organisations, were mentioned by interviewees as positive factors, in particular with the UN and the Council of Europe. Also highly rated were operational cooperation and joint publications such as the Legal Handbooks

which are produced in cooperation with the European Court of Human Rights.

The FRP and collaboration between FRA and civil society were also mentioned as positive and important by various interviewees.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Efficiency

The Agency has achieved considerable efficiency in its operations. There seems to be a clear

acknowledgement by internal stakeholders that the recent changes to the organisational structure were

warranted and could offer both a streamlining of internal processes with the creation of the Corporate

Services department and an improvement to the Agency’s communicatio n and disseminations activities

through the creation of the Fundamental Rights Promotion Department.

In order for the Agency to be fully aligned with the Common Approach and to be as efficient as possible while ensuring its independence, a change in the Founding Regulation would be necessary.

There seems to be an acknowledgement that the Agency is at the limit of the human resources necessary to achieve its objectives, with respect to areas such as field deployments and accompanying research, and communication and dissemination. Additionally, while a lot has already been achieved, findings indicate that the Agency could further improve the way it prioritises human resources and the dissemination of the methodology for prioritisation across the Agency.

However, there is little consensus over whether the Agency’s financial resources are sufficient to achieve its objectives. Civil society and international organisations have discussed their desire for the Agency to devote more resources to cooperation activities and more nationally targeted activities in order for the Agency to increase its impact at the national level, even though this was understood to be limited by the availability of resources.

At this stage, the Agency seems to be reporting in accordance with Commission guidance and templates, providing satisfactory accounts to the European Court of Auditors throughout the programming period.

The Agency has achieved significant effects at a low cost of intervention for the EU Institutions and national authorities in the production of research and analysis across the EU’s Member States and offers substantial value for money for the EU.

Recommendations

The following recommendations for actions are based on the findings and conclusions of the evaluation and are divided according to their evaluation criteria. They should be read in combination with the recommendations of the 1 st independent evaluation of the Agency. The recommendations are addressed to the FRA, except when expressly noted.

Overall

 The Agency’s work is clearly highly regarded by stakeholders. The Agency should continue doing

what it does.

Relevance

 To the Commission, the Council and the Parliament: The Agency’s Founding Regulation should be

modified in order to bring the wording in line with the Post-Lisbon reality and thus increase legal

clarity. A revised wording of the regulation could stress the importance of the Charter as a now

legally binding standard and make explicit that the regulation covers judicial cooperation in criminal

matters.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

 The Agency should conduct further stakeholder consultation on the nature of the new activities that

stakeholders reported requiring from the Agency, regardless of whether these activities are in the Agency’s current mandate, in order to more accurately capture the exact fundamental rights needs of the Agency’s stakeholders, which could potentially act as greater evidence for a revision of the Agency’s mandate.

Effectiveness, Utility and Impact

Regarding meeting the needs of stakeholders:

 The FRA should continue to be responsive to requests (particularly on current emergencies like the

migration crisis) whilst not neglecting on-going research projects.

 The need to plan research projects in advance should continue being balanced with ad-hoc needs

(e.g. allow re-scoping of long-term research projects to grant researchers the time to work on adhoc needs).

 FRA should focus on gaining more visibility at the national level. One way is to engage more closely

with those NGOs and NHRIs that could build a link to authorities.

 The Agency should continue to set clear and realistic targets on both thematic and strategic levels.

There could be a discussion on whether the scope of the thematic areas shall be limited (e.g. by combining one or more thematic areas in one) in order to limit the scope of FRA’s activities in light of resource limitations.

 On a methodological point, it would be useful to structure the mid-term strategic review differently

in order to more easily detect how FRA is doing in respect to each objective (e.g. providing a clear comparison between target and progress).

 The Commission could look into the relevance, appropriateness and need of the MAF that currently

defines the thematic areas for a period of five years and through a decision adopted not by the Agency but the Council of the European Union.

 To the Commission, the Council and the Parliament: the FRA should be permitted to initiate

research in the area of police and judicial cooperation in criminal matters, and social rights, in order to meet the needs of stakeholders at the national and EU level.

Regarding the Agency’s communication:

 The Agency should focus on increasing its impact at the national level by building country-specific

knowledge and expertise on Member States, in particular those that warrant specific attention at any point in time.

 FRA should focus on those stakeholders in its dissemination strategy that have a multiplying effect

(e.g. more interaction with media in addition to the work on social media, which reaches mainly individuals already aware of FRA’s activities).

 FRA should clarify with its diverse set of stakeholders who its key audience is – namely, EU

Institutions and Member States’ governments – which would serve to explain its different levels of engagement with various stakeholders that include and go beyond these groups.

 The Agency should continue to capitalise on the wealth of research and analysis it has accumulated

by ‘re-packaging’ it in publications and other activities that could have a wider impact, especially at the national level.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Added value

The European Parliament and Council should ensure that the Agency can:

 Continue to focus on elements that makes it distinct from other Fundamental Rights organisations

and adds value to its stakeholders, in particular:

o The collection and analysis of comparable data;

o Support to policy-makers in developing fundamental-rights compliant legislation;

o Real-time and practical assistance and advice based on support and activities undertaken in the field (such as the Agency’s work in migration hotspots);

o The Agency’s independence should continue to be guaranteed.  Provide further forums for discussions on fundamental rights involving relevant stakeholders from

across the EU Member States, as well as with additional stakeholder groups.

Coordination and Coherence

The Agency should:

 Continue the complementary collaboration with EU/national policy stakeholders, international

organisations and civil society, and investigate the complementarities that were highlighted by a handful of stakeholders in the evaluation.

 Where possible within FRA’s remit, formalise greater cooperation between international

organisations with cooperation agreements as well as the production of more joint outputs.

 Continue expanding the level of cooperation with civil society.

Efficiency

The Agency should:

 Put the business case forward for further investment in communication and in dissemination

activities in order to maximise the impact of the Agency’s activities.

 Clearly delineate and communicate the roles and responsibilities of staff and the new departments

amongst the Agency’s staff members.

 Following and endorsing the recommendation from the Agency’s first external evaluation, create

the title of Deputy Director with the task of day-to-day management of the Agency, in order to prioritise the Director’s activities towards leadership, outreach and cooperation.

 Sufficient resources should be provided for research activities corresponding to new requests

received annually by EU Institutions for more research evidence on fundamental rights issues;

 The Agency should be given an increase in its human and financial resources, in order to enable it to

reach its critical mass in terms of impact, which will allow the Agency to respond effectively to the increased demands placed on it with respect to data collection, having a presence on the ground in

response to fundamental rights emergencies, and communication and dissemination.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Case study 1 – Violence Against Women survey

Background and introduction

Background

Violence against women is a key human rights problem that impacts across all EU Member States, affecting and undermining core fundamental rights such as dignity, access to justice and gender equality (amongst others). In 2009, a European Parliament Resolution called for the ‘collection and compilation by the FRA of reliable, comparable statistics on all grounds of discrimination, including discrimination against national minorities, and for the equal treatment of those different grounds, including comparative data on violence against women within the EU, and their publication in readily

understandable form’. 218 This call from the European Parliament was further mentioned by the Council of the EU in its conclusions on the eradication of violence against women (8 March 2010). 219 In recent years, the EU has also taken firm positions on the need to eradicate violence against women and has funded specific campaigns and grass-roots projects to combat it. Such positions are particularly reflected

in:

 The European Commission Communication on a Strategy for equality between women and men

2010–2015, affirming that ‘there are many forms of violence that women experience because they are women. These include domestic violence, sexual harassment, rape, sexual violence during conflict and harmful customary or traditional practices such as female genital mutilation, forced marriages and honour crimes. The Action Plan to implement the Stockholm Programme puts emphasis on the protection of victims of crime, including female victims of violence and genital

mutilation, and announces a comprehensive EU strategy on gender-based violence’; 220

 The Council conclusions on the Eradication of Violence Against Women in the European Union,

urging Member States to ‘continue and update or develop national strategies for tackling violence against women in a comprehensive way, devote appropriate resources to preventing and

combating violence against women and provide assistance and protection to victims’; 221 and

 The Commission Staff Working Document on Strategic engagement for gender equality 2016–2019,

affirming that ‘gender-based violence is still widespread and can take many forms: one woman in three has experienced physical or sexual violence, 5% have been raped since the age of 15 and 20%

have experienced online harassment’. 222

In this context, the FRA launched, in its Annual Work Programme 2010, the project ‘Violence Against women: an EU-wide survey’, aiming to provide the EU and the Member States, for the first time, with comparable data which can be used to develop policies to combat violence against women, protect the

victims, and raise awareness on the issue, both among the general public and among specific groups of

218 P7_TA (2009)0090 (on the Stockholm programme)

219 FRA Annual Work Programme 2011

220 COM (2010) 491 final i, available at http://eur-lex.europa.eu/legal-content/EN/TXT/?qid=1396540108305&uri=CELEX:52010DC0491

221 Council conclusions of 8 March 2010 on the eradication of violence against women in the European Union, available at

https://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/en/lsa/113226.pdf

222 Commission staff working document ‘Strategic engagement for gender equality 2016–2019’ SWD (2015) 278 final, available at

http://ec.europa.eu/justice/gender-equality/files/documents/151203_strategic_engagement_en.pdf 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

practitioners working with victims of violence. The project was built upon the experience of the Agency in victimisation survey research (e.g. EU-MIDIS survey) as well as in-house expertise on violence against

women survey research, and its main objectives were:

 To provide the first EU-wide dataset on the extent, nature and consequences of violence against

women, as reported by women, which can be used to inform policy and action on the ground;

 To highlight the manifestation of gender-based violence against women as a fundamental rights

abuse in the EU.

The project touched upon three FRA MAF thematic areas 223 and its specific objectives fit within the strategic framework of the FRA, specifically addressing the following Agency strategic objectives:

 Assist the EU and its Member States in decision-making by providing quality and relevant data, facts

and opinions.

 Inform target audiences through awareness-raising activities and use of data collected in the field

that provides factual evidence.

 Develop methods and standards to improve comparability, objectivity and reliability of data at

European level, in cooperation with the Commission and Member States.

The research also aimed to offer useful guidance to Member States in their ratification and implementation of the Council of Europe Convention on violence against women and domestic violence (Istanbul Convention) and to offer policy-makers and other stakeholders in the EU, and at national level, necessary evidence for developing legislation and policies to address violence against women through

prevention of incidents, protection of victims and prosecution of offenders.

Prior to the launch of the project, there was no EU-wide survey on violence against women and the only data and statistics available referred to stand-alone surveys carried out in some EU Member States, which mostly produced non-comparable data (with the exception of three EU Member States which have produced directly comparable data based on a standard International Violence Against Women Survey questionnaire). Within DG Research’s Sixth Framework Programme, research under the heading of ‘Coordination Action on Human Rights Violations (CAHRV)’ had attempted a comparative re-analysis of five independent violence against women surveys in EU Member States in order to explore possibilities for comparison of existing surveys. However, the results identified a number of problems

with trying to compare findings from different survey instruments. 224

Introduction to the case study

This case study aims to analyse the impacts of the FRA Violence Against Women Survey, assessing in particular to what extent the activities undertaken had an impact on national and EU policies and practices, as well as on increased dialogue, coordination or cooperation within the EU Institutions and between EU Institutions and Member State authorities. Due to the limited availability of impact-related data, the analysis presented in this case study does not include reactions of the scientific community on the results of the Violence Against Women Survey and does not encompass judgements on the scientific

value of its findings. However, the Agency’s own Scientific Committee and a panel of external academic

223 Discrimination based on sex, race or ethnic origin, religion or belief, disability, age or sexual orientation and against persons belonging to

minorities and any combination of these grounds (multiple discrimination) – The rights of the child, including the protection of

children – Access to efficient and independent justice

224 FRA Violence Against Women, 2012 – Project Concept Fiche

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

and non-academic experts on survey research and the field of research of violence against women were

engaged over several meetings in overseeing and discussing the survey’s development.

On the basis of the project fiches published in the FRA Annual Work Programmes and the FRA Strategic

Plan, the project was anticipated to produce the following impacts, duly reported on in this case study:

 Relevant stakeholders received reliable, relevant and high quality statistical information on violence

against women in EU Member States;

 Best practices and methodologies were shared with relevant stakeholders;

 Research findings influenced the development of policies and legislation at EU and/or national level;

 There is an increased compliance with the Council of Europe Convention on violence against women

and domestic violence (Istanbul Convention);

 Relevant EU and Member State institutions, bodies and agencies and/or civil society increasingly

addressed and promoted campaigns and surveys on VAW.

However, due to the limited availability of data, the following impact cannot be considered in the scope of the case study as it refers to long-term and aspirational effects on project beneficiaries and,

therefore, require a broader qualitative and quantitative research effort.

 There is an increase of the levels of prevention of incidents, protection of victims and prosecution of

offenders through new policies and legislation.

Figure 28 presents the full intervention logic of the FRA Violence Against Women Survey project, including its thematic context, MAF strategic and thematic objectives, rationale, inputs/ resources,

activities, outputs and expected impacts.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 27: Violence Against Women Survey Intervention Logic

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Activities / resources

The FRA Management Board allocated funds for the Violence Against Women survey pre-testing during the December 2009 Management Board Meeting. The development of the survey started in January 2010, with a desk research phase and consultation process as well as the collection of experts’ inputs concerning the content of the survey. Pre-testing activities (cognitive interviews combined with behavioural coding, and focus group discussions) took place between January and July 2011. 225 The contract for the full-scale 28-country survey was signed in December 2011, and data collection activities were carried out over 12 months including sampling, development of fieldwork documents (based on the FRA questionnaire), translation of fieldwork documents (questionnaires, introductory letters, training materials), selection and training of interviewers, fieldwork monitoring, data quality checks, data management and reporting. The data collection phase involved face-to-face interviews with 42,000 women in all 28 EU Member States, with an average of 1,500 interviews per Member State, and respondents were selected based on random sampling. The results of the survey are representative of the experiences and opinions of women who are between 18 to 74 years old and live in the European Union. Overall, project timelines, objectives, and financial and human resources outlined in the Project Concept Fiche were duly respected. However, changes were made with respect to the expected outputs

and publications, as further explained above.

The total budget committed by the FRA for the Violence Against Women project was €4,044,800, distributed throughout the period of its implementation as follows:

 AWP 2010: €249,500 (pre-testing);

 AWP 2011: €1,795,300 (data collection, part 1);

 AWP 2012: €1,550,000 (data collection, part 2);

 AWP 2013: €270,000 (publication);

 AWP 2014: €160,000 (publication, follow-up);

 AWP 2015: €10,000 (follow-up);

 AWP 2016: €10,000 (presentations and events).

The core FRA team that worked on the Violence Against Women Survey project included one Head of Department (15% of time over 2010–2015), one Project manager (70% of time over 2010–2015), two Senior project officers for the analysis of results (30% of time over 2010–2015), and one Communication and stakeholder engagement officer (20% of time over 2010–2015). Further FRA staff members contributed to the consultation meetings, translations of fieldwork documents, training of interviewers and interviewing oversight in different Member States, editing the research outputs, media engagement

and communication of results.

Outputs

In January 2013, the FRA started analysing the data and drafting the main project outputs, which included a comprehensive results report, a technical report, a summary report and a factsheet as well

225 The FRA published a call for tender for the pre-testing in August 2010 and the contract was signed in December 2010.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

as the interactive online data explorer. The final outputs are not fully in line with the list of planned publications presented in the Project Concept Fiche, which consisted of five short reports (including two handbooks). However, it is assumed that such change was driven by the quality and type of research data acquired by the FRA at the end of the data collection phase, and that the type of reports chosen enabled the FRA to report on findings in a way that ensured consistency and coherence with the type of research performed. Based on the survey’s findings, the final comprehensive report by FRA made a number of recommendations, including, amongst others:

 The findings of the survey need to be included in future strategies on equality between women and

men;

 The EU’s justice and home affairs policy should include violence against women in its framework of

the EU’s responses to crime and criminal victimisation;

 Member States should apply the Victims’ Directive to victims of gender-based violence;

 Member States should ratify the Istanbul Convention and the EU should try to sign and ratify the

Istanbul Convention. 226

The full results report and the technical report were made available only in English, while the summary report and a factsheet have been translated into 22 languages, and the online data explorer is available in English, French and German. The survey results were launched at an event organised in the European Council premises on 5 March 2014. The launch was followed by dissemination and awareness-raising activities of project results in over 20 EU Member States. This included presentations in seminars, conferences and other events related to violence against women, as well as meetings with relevant government representatives, service providers, academics and civil society organisations.

Impacts

The project has, overall, achieved all the objectives outlined in the relevant project fiches of the FRA Annual Work Programmes. The survey filled a gap identified by European Institutions and Member States of a persistent lack of comparable data that decision-makers need in order to shape informed, targeted policies. For the first time, comparable data on women’s experiences of violence was made available to policy-makers and practitioners in all EU Member States, particularly for the development of policies and other measures to combat violence against women, also enabling comparisons of the

extent of violence against women in different Member States. 227 Data collected from stakeholder

interviews, surveys, statistics, and through a literature review, show that the FRA Violence Against Women Survey generally produced impacts both at national and EU level (see below for a detailed assessment).

Short-term impacts

The FRA has carried out an effective awareness raising and dissemination campaign of the outputs of the Violence Against Women Survey. The results of the survey were presented in 44 events in 2014, 12 in 2015 and 16 in 2016, including international conferences and seminars, working groups (e.g. Eurostat working group on crime statistics, Council of Europe Committee (GREVIO) responsible for monitoring

226 European Pparliament (2016), The Issue of Violence Against Women in the European Union, available at

http://www.europarl.europa.eu/RegData/etudes/STUD/2016/556931/IPOL_STU(2016)556931_EN.pdf

227 FRA website, http://fra.europa.eu/en/project/2012/fra-survey-gender-based-violence-against-women (accessed on 02 October 2017)

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

implementation of the Istanbul Convention, Council of Europe drafting group on female genital mutilation and forced marriage, UNFPA technical advisory committee on strengthening regional and

national capacities for measuring violence against women, German Bundestag hearing on the survey’s results, UK Parliament’s Joint Committee on Human Rights (JCHR), CEPOL webinars – as well as several

roundtables in other EU Member States.

The launch of survey results was the biggest launch of a FRA product in the history of the Agency, with at least 1,500 news articles and broadcast reportages during the first four days across EU member States

and the world. Media statistics show that media in all EU Member States reported on the survey and, on the launch day, all EU media with the widest reach covered the report in their headline news. 228 The FRA

Director, Heads of Department, Experts and Spokespersons gave a total of 73 interviews to media from 16 EU Member States and international outlets. Moreover, within the first seven days, FRA’s website registered very high traffic, with around 12,000 publication page views and over 8,000 downloads of the

main results report. 229 The FRA has also implemented a targeted press release strategy involving communication ‘multipliers’, social media and other communication material as well as direct emailing.

The press release was sent in all EU official languages to FRA’s media contacts, recording higher interaction percentages in all countries in comparison to previous releases, with highest rates of

interactions recorded in Germany, Austria, Belgium, France, the Netherlands, Sweden, Romania, Portugal, Bulgaria and Slovakia. In particular, the press release via direct emailing has shown the most

significant impact on the media, with 2,268 journalists directly reached around Europe (sent in 21 languages) and significant increases (compared to 2013) of related clicks on the report or online media

pack published on FRA’s website (e.g. +46% for the English version), showing that the Survey results have reached a higher number of journalists compared to previous FRA publications.

As outlined in Figure 29, data on downloads of the Violence Against Women Survey outputs also shows that the results have reached a large number of stakeholders. The main products downloaded from the FRA website in 2014 were the main survey results (32,598 downloads) and the factsheet (26,221

downloads), with clear evidence of a long-term interest of stakeholders in the survey results, particularly in the case of the main results report, which was downloaded almost 32,000 times even in 2016. From

this perspective, the survey has achieved exceptional dissemination results in comparison to other FRA publications, as only few single FRA product shows similar trends in terms of copies downloaded two years after the publication of the report (when compared to the Handbook on European nondiscrimination

 law, Handbook on European law relating to asylum, borders and immigration, and Handbook on European data protection law, which have achieved a lower number of downloads

overall).

Furthermore, the Violence Against Women main results report was the most downloaded FRA publication (within the first week of publication) in the past four years, with over 8,000 downloads,

followed by Severe labour exploitation: workers moving within or into the European Union with 1,622 downloads, Surveillance by intelligence services: fundamental rights safeguards and remedies in the EU

with 1,492 downloads, Professionally speaking: challenges to achieving equality for LGBT people with 1,373 downloads and the Fundamental Rights Report 2016 with 1,327 downloads. 230

228 The findings appeared on the front page of several EU newspapers and were included in the morning and evening news of national radio and

television stations across the EU. Amongst the countries were most coverage was recorded, the highest impact was in Germany with

58 media reporting on the survey, followed by Spain with 38 media and Italy with 26.

229 FRA Media analysis – VAW report and survey 2014

230 Ibid.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 29: Violence Against Women Survey outputs downloads

Source: VAW survey publications: download statistics

Long-term and Aspirational impact

The evidence collected through this case study suggests that research findings from the Violence Against Women Survey have been frequently used by EU and national-level policy makers and practitioners, and possibly influenced the development of policies and legislation. As outlined in Figure 30, the majority of EU Officials (57%) and almost half of the non-EU institutions stakeholders surveyed, such as representatives of national ministries, gender equality commissions and ombudsmen, reported having used scientific surveys undertaken by FRA, including the Violence Against Women Survey, often or very often during the past five years. Additionally, all European Commission and European Parliament authorities and the large majority of the non-EU institutions (85%) affirmed that such FRA outputs were useful or very useful for their work between 2013 and 2017. Commission Officials responding to the survey and interviewed in the context of the external evaluation of the FRA stated that the Violence Against Women Survey has been very influential in their work and capacity to promote fundamental rights across the European Union, as it showed in a statistical and quantifiable way the extent of the problem in the EU, pushing European Institutions to be more active in eradication of gender-based violence. As outlined in the previous section, the use of FRA scientific surveys, including the Violence Against Women Survey, for policy-making purposes has been high amongst relevant policy-making stakeholders, and the survey report has achieved a high number of downloads. Therefore, it can be reasonably assumed that the survey has been used frequently by policy-makers and has influenced

policy-making at EU and national level.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Figure 30: How frequently have you used the FRA outputs listed below in the past five years?

Source: Optimity – Survey of EU Institutions and non-EU institutions

Furthermore, stakeholders reported that the FRA, through its Violence Against Women Survey, had an impact on the development of EU legislation, policies and practices and that the survey results were a driving factor in the EU signing the Council of Europe Convention on preventing and combating violence against women and domestic violence (Istanbul Convention) in 2017. The Istanbul Convention, presented in 2011 and entered into force in August 2014, was the first European Convention on violence against women, including domestic violence, and is also legally binding. It sets out minimum standards for Member States signing and ratifying the Convention on the prevention of violence against women, the protection of victims and the prosecution of perpetrators, and calls for signatories to take the appropriate legislative steps to this end. 231 The FRA survey is mentioned in the European Commission Proposal for a Council Decision on the conclusion, by the European Union, of the Council of Europe Convention on preventing and combating violence against women and domestic violence, concluding that ‘both Member States and the Union shall become Parties to the Convention’ and approving the Convention on behalf of the Union. 232 Out of the 44 States that acted on a legislative basis on the Convention, 11 EU Member States ratified it after the publication of the FRA Survey. Although the available evidence does not allow a direct link to be made between the FRA Survey and the ratification rate of the Istanbul Convention, data suggests that discussions at national and European level on the results of the Violence Against Women Survey have facilitated the social and political dialogue on this theme, particularly on aligning national laws with the Council of Europe Istanbul Convention. For

example, the FRA Survey have been mentioned in:

 National parliamentary discussions and action plans (AT, DE, ES, FR, IT, PT, UK);

 National legal documents (ES, FR);

231 European Parliament (2016), The Issue of Violence Against Women in the European Union, available at

http://www.europarl.europa.eu/RegData/etudes/STUD/2016/556931/IPOL_STU(2016)556931_EN.pdf

232 COM (2016) 109 final i, available at http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52016PC0109&from=EN

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

 European Parliament Resolution of 10 March 2015 on Progress on equality between women and

men in the European Union;

 17 reports, positions and/or legal documents of the European Parliament, European Commission

and Council of the European Union;

 18 reports, opinions and action plans of Non-Governmental Organisations and networks (e.g.

Amnesty International, Women’s Aid, European Woman Lawyers’ Association, Human Rights Watch), International Organisations (e.g. Council of Europe – Commissioner for Human Rights, UN Women, UN Economic Commission for Europe, World Bank, European Institute for Crime

Prevention and Control, UNESCO) and EU – Parties (European Left – FEM).

Furthermore, the FRA presented the findings from the Violence Against Women Survey at the 8th meeting of the Council of Europe’s Group of Experts on Action against Violence against Women and Domestic Violence (GREVIO), which is responsible for monitoring the implementation of the Istanbul Convention. The Council of Europe body was interested to hear about the range of FRA’s work on violence against women, including the survey, and the FRA will assist the Council of Europe in their

monitoring work through exchange of information.

Finally, the Survey contributed to provide Member States with a well-documented methodology on how to collect data on violence against women and report on EU Council indicators. The survey has developed a set of comparable indicators at European and national level which can be used as a basis for the development of national statistical instruments. For example, Spain has adapted the questionnaire of its Macro-Survey on Violence against Women 2015 in order to meet the quality requirements recommended by the Fundamental Rights Agency, which the previous macro-surveys had not yet incorporated. 233 At European level, the FRA has recently (2017) taken part in a Eurostat task force, together with the European Gender Institute (EIGE), the European Commission and independent experts, set up to start developing a survey on gender-based violence in the EU. The FRA presented its survey methodology and questionnaire, and introduced considerations for the development of indicators based on survey data. At the last task force meeting (June 2017) it was decided that the survey questionnaire will follow the approach of the FRA survey on violence against women and will collect data on the extent and nature of physical, sexual and psychological violence between intimate partners, violence by other perpetrators, sexual harassment, and stalking and violence in childhood on

both women’s and men’s experiences of violence.

The OSCE is also duplicating FRA’s Violence Against Women Survey in 10 non-EU Member States, having started the process towards duplication shortly after the launch of the FRA’s survey in 2014. The FRA is part of the OSCE steering committee that oversees the development of this survey. Outside of the

European context, a university in Japan has also replicated FRA’s survey.

Due to the limited availability of data, it was not possible to analyse the assumed impact of the FRA Violence Against Women Survey on possible increased of levels of prevention of incidents, protection of

victims and prosecution of offenders in EU Member States.

Conclusion

The analysis of the impacts of the FRA Violence Against Women Survey presented above outlines that the project has overall achieved all the planned objectives and produced relevant impacts, particularly

233 Delegación del Gobierno para la Violencia de Género, Macroencuesta de Violencia Contra la Mujer 2015

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

at short-term and long-term level. Survey results reached a large number of people in the European Union, both amongst relevant stakeholders (policy-makers, practitioners and non-governmental organisations) and the general public, and contributed to fostering political and social debate towards the development of policies and other measures (including legislative measures) to combat violence

against women.

Furthermore, the survey produced, for the first time ever, comparable data on violence against women in the EU, as well as a methodology 234 and set of indicators which have already been used as a reference for the development and implementation of similar exercises at European and national level, thus contributing to an increase of efforts by EU and Member State institutions and bodies in monitoring the levels and trends of gender-based violence. Finally, results of the survey have contributed, to a certain extent, to fostering compliance with and adhesion of EU Member States to the Council of Europe Convention on preventing and combating violence against women and domestic violence, and have certainly provided the European Union with key data which supported the debate leading to the signing

of the Convention in June 2017.

234 Note that the scientific quality of the methodology developed and the scientific value of its findings could not be assessed as part of this case

study. Nevertheless, the FRA Scientific Committee closely followed the development of the methodology used for the survey and a

panel of external experts supported the entire project life cycle.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Case Study 2 – FRA activities in hotspots

Background and introduction

This case study concerns the deployment of FRA staff to Greek hotspots between 2016 and 2017, and the visits to the Italian hotspots in 2017, in the context of the refugee crisis.

Since 2010, FRA Annual Work Programmes have included activities in support of Frontex, which were subsequently extended to include activities in support of other EU agencies and institutions. Following a request from DG Justice in 2015, the Agency started to publish regular updates on the refugee situation in the EU, providing coverage of the nine EU Member States most affected, which was subsequently extended to 14 EU Member States. In addition, FRA has been providing fundamental rights expertise to the European Commission, Frontex and EASO on diverse aspects of the operations in Italy and Greece, since the inception of the hotspots approach proposed in the European Agenda on Migration of May 2015. Although FRA welcomed the benefits of the hotspot approach from a fundamental rights point of

view, it also was attentive to its risks.

In January 2016, the FRA Director visited Greece, Italy and Germany to better understand the situation on the ground with a view to determining whether FRA’s response to the asylum and migration situation was adequate. In this regard, the FRA observed a number of protection gaps that required attention

when further operationalising the ‘hotspot approach’.

As a result, from the beginning of April to the end of August 2016, FRA deployed staff to Greece to provide more operational fundamental rights support directly to EU actors involved in the operation of the hotspots. From September 2016 onwards, the Agency redirected its resources from Vienna to implement specific activities to address some of the remaining gaps in Greece. In November 2016, the Agency and the Hellenic Ministry of Migration Policy signed a Memorandum of Understanding in order to facilitate and deepen the cooperation between the Agency and the competent national authorities in Greece. From December 2016 to September 2017 FRA visited several hotspots to provide a number of trainings and workshops, as well as to participate in meetings with a number of relevant actors on the ground in Greece. In addition, FRA has been exploring how best to contribute to the situation in Italy

and visited hotspots in Italy in September 2017 in this context. 235

As discussed, this case study is aimed at better understanding the impact of the activities undertaken by the Agency in the Greek and Italian hotspots between April 2016 and May 2017, in the context of the on-going refugee crisis. This particular activity was selected for the case study due to the significant importance of the issue this project sought to address and because this activity fell outside the Agency’s more traditional core activities. This case study can therefore demonstrate the relevance and flexibility of the Agency in addressing major emerging fundamental rights issues in the EU, in addition to its

impact.

The methodology for this case study consisted of a document review, undertaking targeted interviews as well as gathering relevant data from the evaluation’s large-scale interview programme.

235 Interview FRA Freedoms and Justice Department October 2017.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Intervention Logic

Needs that the case study aimed to address

The needs that the Agency’s activities in the Greek and Italian hotspots aimed to address were the needs of the EU Institutions and national authorities relating to their role in ensuring the protection of the fundamental rights of migrants and asylum seekers arriving in Greece and in Italy, in the context of the EU hotspot approach.

Concept note: The hotspots entail an unprecedented on-the-ground presence and direct engagement of EU actors in managing asylum and migration flows. This has direct fundamental rights implications for the Union itself. In light of these exceptional circumstances, the Agency plans to deploy staff members to

Greece to provide real-time support to EU actors on the ground. 236

In its Interim assessment of August 2016, the FRA identified a number of protection gaps or challenges that require attention, namely:

 Child protection: this is one of the most serious protection gaps in the hotspots with child

detention, lack of adequate accommodation, exposure to risk of violence and exploitation, ineffective guardianship system and a lack of child protection staff. Serious child protection issues also persist on the mainland, including in the context of relocation.

 Gender issues and in particular gender-based violence: overcrowding, inadequate camp

infrastructure (e.g. risks for women and girls to access sanitary facilities at night in some hotspots); the absence of effective policing in the hotspots, non-existent or non-implemented standard operating procedures to deal with reports of gender-based violence and impunity of perpetrators serve to expose people (particularly women and girls with respect to gender-based violence) in the hotspots to risks of violence and abuse, particularly at night, when humanitarian staff are not present.

 Identification of vulnerable people during initial registration: identification of vulnerabilities

during initial registration procedures usually carried out with substantial Frontex support remain very weak, with limited awareness (and sometimes the use of inadequate forms). Referral and follow-up needs also to be improved.

 Respect for safeguards for deprivation/restriction of liberty, return and readmission: significant

gaps have been identified by FRA in relation to provision of information, notification of decisions and access to effective remedy.

 Access to legal aid in asylum cases: the limited capacity to offer legal aid to asylum seekers and

migrants puts in question the right to an effective remedy against a negative asylum (admissibility) decision with significant impact on EASO’s work in the hotspots.

236 FRA, Concept note: FRA ad-hoc presence in Greece to support EU actors on the ground with fundamental rights expertise and advice: April–

September 2016 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Objectives

The main objective of the FRA activities was to assist and provide fundamental rights expertise to EU actors and national authorities to address operational challenges in the hotspots in Greece and Italy, to

support them to carry out their tasks in full compliance with the EU Charter.

The project was not explicitly mentioned in the Annual Work Programme (AWP) for 2016. However, the project falls within the remit of the Thematic objective ‘To promote effective fundamental rights safeguards in the development and implementation of EU policies in the field of immigration and integration of migrants, visa and border control and asylum’, under the Project 5.1.2. ‘Inter-agency

cooperation and other activities in the areas of borders, immigration and asylum’.

The FRA’s AWP for 2017, which is included in the FRA Programming Document 2017–2019, does explicitly plan the activities in the hotspot, namely as ‘Project A 7.5. – Providing fundamental rights expertise to address operational challenges’. The project description also refers to activities undertaken

in 2016.

With regard to the activities undertaken in Greece, the FRA concept note, the 2017 AWP and the interim assessment carried out by FRA in August 2016 include the following objectives, which are in line with

the needs identified:

 to provide rapid access to an EU fundamental rights advice capacity on the spot, including the

sharing of promising practices;

 to help FRA calibrate its assistance, including its training support, so that it is tailor-made to the

practical challenges experienced on the ground;

 to map and observe processes at hotspots and operating actors, identifying relevant fundamental

rights challenges for follow-up by FRA;

 to provide regular feedback to national authorities, Frontex, EASO and the European Commission

staff on the ground on the main issues identified and discuss possible actions which could be taken to overcome them;

 to promote awareness and implementation of child protection safeguards and measures to keep

children safe (special focus for 2017);

 to promote awareness of measures to protect vulnerable persons (special focus for 2017).

The objective of activities in Italy in 2017 were mainly to assess the evolving migration situation in the Central Mediterranean and the need for further FRA support. Specific objectives for targeted support in Italy have not been laid down in any planning document. Instead these were discussed bilaterally with

the Italian authorities and the European Commission in late 2016.

Inputs

Human Resources:

Table 22 provides an overview of the human resources planned to staff this action in Greece, as well as those actually used.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Table 22: Project resources planned and actual

April–Sept 2016 September–December 2016 2017

Planned Actual Planned Actual Planned Actual

 1 FRA  1 FRA N/A N/A

Staff in Greece N/A

staff in staff in and Italy: 1.3

Greece; Greece;

 1  1 support

interpret staff in er for Greece;

field  1.5 FRA missions; Staff as

 0.5 FRA backstop

Staff as in Vienna;

backstop  

in Vienna;

 0.25 FRA

staff for input (ADM and specialise d).

TOTAL: 2.75 TOTAL: 3.5 1.3

All the above resources were to work under the guidance of the Head of Sector, Asylum, Migration and Borders in the Freedoms and Justice Department. However, on the basis of an interim assessment, at the beginning of April 2016 FRA deployed one member of staff on a rotation basis to Greece. This person was assisted by a junior support staff. In this context, the Agency set up a rotating roster of six members of staff with the relevant expertise who were drawn from three different departments, to be deployed to Greece. Moreover, the assessment reports that approximately 1.5 staff members in Vienna worked on Greece-related issues, constituting twice as many human resources than initially planned.

No staff were deployed to Greece or Italy on a more permanent basis from September to December 2016. FRA estimates that 1.75 staff members worked on the activities from Vienna and Athens during this time (excluding the drafting of the hotspot opinion requested by the European Parliament).

237 FRA, Concept note: FRA ad-hoc presence in Greece to support EU actors on the ground with fundamental rights expertise and advice: April–

September 2016

238 FRA Annual Work Programme 2017, p. 76.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Financial:

The estimated costs for a six months deployment between April and Sept 2016 and for 2017 are as shown in Table 23.

Table 23: Planned and actual project costs

Costs April–Sept. 2016 Sept.–Dec. 2016 2017 239

Planned 240 Actual Planned Planned

DA + hotel €39,600 N/A N/A

International flights €4,000 N/A N/A

Internal flights €6,000 N/A N/A

Other (e.g. taxi, interpreter) €10,400 N/A N/A

Total €60,000 €57,000 N/A €230,000

Activities

Activities undertaken in Greece

Activities undertaken in Greece from April to July 2016 mainly related to visits to the hotspots, collecting data and information and reporting back to national authorities, Frontex, EASO and the European Commission, participating in meetings and facilitating on-going processes to address gaps, as well as providing training. After July 2016, FRA staff also undertook capacity-building activities from Vienna and Athens in the specific fields of child protection and vulnerable people, as well as policing, returns and

242

readmissions. Across the entire 2016–2017 period, the types of activities that the FRA has undertaken

in Greece can be categorised in four types of activities (also further detailed in the table below):

 Visits to the hotspots and related activities such as reporting back the observations of FRA in the

hotspots in regular letters sent to the European Commission Frontex and EASO;

 Participating and contributing in meetings with the EU actors and national authorities;

239 FRA Annual Work Programme 2017, p. 76.

240 FRA, Concept note: FRA ad-hoc presence in Greece to support EU actors on the ground with fundamental rights expertise and advice: April–

September 2016

241 According to the Interim Assessment of 2016, by mid-August FRA will have spent 19 weeks in Greece with additional missions to attend

specific events. This amounts to some EUR 55,000 in mission costs and estimated EUR 2,000 for communication.

242 FRA, Final note to the attention of the European Commission, Summary of FRA’s findings from its temporary deployment in Greece: April–

August 2016, Planned FRA actions for 2016 to address needs identified in Greece, 31 August 2016.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

 Providing advisory services to national authorities on particular issues, such as the standard

operating procedures;

 Capacity building, including providing training and workshops to national authorities and

developing and disseminating materials.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Visits to hotspots & Meetings Advisory Capacity building

related activities

 Observing and  Participate in coordination  Review screening forms  Providing trainings and workshops:

collecting data and meetings and other meetings: used for registration (July o

information o Closed meeting with 2016) Workshop on guardianship sharing promising practices from other EU Member States (July 2016)

 Providing Hellenic Police, Greek  Prepare an overview of o Tailored training on return/ readmission

fundamental rights Ombudsman and others main child protection gaps monitoring for Greek Ombudsman’s staff (July expertise on the to identify ways to on the five islands and 2016)

spot improve respect for identify actions to address

 Regular letters to fundamental rights in these (March 2017)

o Provide training to Reception and Identification Service staff in Kos, Leros, Chios and Samos (2017)

the Ministry of detention, removal/  Support the Reception and Migration Policy and readmission (July 2016) Identification Service with o Workshop on preventing and responding to selfto

 the European o Coordination meetings by the finalisation of standard harm in the hotspots (May 2017) Commission, the SRSS in Athens operating procedures  Dissemination of relevant FRA materials

Frontex and EASO o EU RTF meeting: for (SOPs) (2017) o

Checklist to map child protection issues in hotspots

 Mapping of services example, FRA contributed  Provide comments to a (July 2016)

for unaccompanied to an agenda item on form for vulnerability o Develop a video on child protection for Frontex

children in hotspots UAM (March 2017) assessment (2017) pre-deployment briefings, together with Frontex

(August 2016) o Meetings between FRA (2016)

Director and Greek o Planning short video for training EASO deployed Minister of Migration staff in Greece linking EASO code of conduct to the

Policy Charter (focus on vulnerable people)

o Translation of FRA police training manual into Greek (August 2016)

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Activities undertaken in Italy

In September 2016, the Agency undertook two missions to hotspots in Italy (Taranto and Pozzallo) in order to better understand the protection challenges and assess the capacity for action by the Agency. Following discussion with the Italian authorities and the European Commission, targeted actions by FRA were agreed upon. Activities undertaken are quite similar to those carried out in Greece and include

visits to hotspots, organising workshops and presentations and participating in meetings.

In 2017, FRA visited the hotspot in Taranto (April 2017), Lampedusa (June 2017) and Pozzallo (Sept. 2017), assisted in a disembarkation operation in Catania and visited an Italian Coast Guard ship and a UK Border Force vessel (part of Frontex operation Triton) involved in search & rescue operations. FRA also

had meetings with the EURTF in Catania.

Secondly, the FRA carried out a workshop in the Taranto hotspot in May 2017 on good practices, to increase awareness of the specific needs of children and vulnerable people with the Italian Ministry of Interior, the European Commission and in partnership with UNHCR. The FRA presented at the EASO’s Consultative Forum Regional Thematic Meeting in September 2017 on ‘EASO Operations: Lessons Learned’ in Trapani. Finally, on 24 October, FRA helped organise and participated in a workshop to share

good practices in the newly created hotspot in Messina.

Thirdly, the Ministry of Interior invited FRA to the hotspot coordination meeting in September 2017, where the Italian hotspot SOPs and a possible revision thereof were discussed.

Outputs

A range of outputs were reported by the FRA between 2016 243 and 2017 244 . The most relevant outputs are the 50 visits to the hotspots in Greece, the sending of monthly letters to the Ministry of Migration Policy, the 36 regular letters to the European Commission, Frontex and EASO, the nine training or workshop sessions provided to the Greek and Italian officials, and the five (informal) meetings or

conferences in which the FRA participated.

In addition, one high-profile output from the work in the hotspots has been the ‘Opinion on fundamental rights in the “hotspots” set up in Greece and Italy’ 245 that the FRA developed at the request of the European Parliament, and presented in the LIBE Committee at the end of 2016. Moreover, a tangible output from all the discussions and meetings with the Greek government is the signing of the

MoU between FRA and the Greek Minister of Migration Policy.

Moreover, FRA provided input to different guidelines, procedures and reports, such as the Greek Standard Operating Procedures.

Finally, the FRA also developed materials, such as the mapping of the child protection services in hotspots, and, more recently a mapping of mechanisms in place to prevent and respond to sexual and gender-based violence. Upon request by the Registration and Identification Service, FRA developed draft

guidelines on the prevention and management of self-harm incidents in hotspots.

243 FRA, MATRIX report 2015-FJ-01 Inter-agency work in the areas of borders, immigration and asylum

Activities and Milestones Report – 5/10/2017.

244 FRA, MATRIX report 2017-FJ-07 Hotspots – Providing fundamental rights expertise to address operational challenges

Activities and Milestones Report – 5/10/2017

245 Available at: http://fra.europa.eu/en/opinion/2016/fra-opinion-fundamental-rights-hotspots-set-greece-and-italy

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Impacts

This section discusses the extent to which the activities undertaken by FRA in relation to the Greek and Italian hotspots had an impact at the national and EU level. The Agency’s activities led to both a direct and an indirect impact on the hotspots.

As discussed above, the FRA activities in the hotspots in Greece and Italy had a range of outputs and results, the main result being an increased level of awareness of fundamental rights issues on the ground by relevant duty bearers including EU Institutions and agencies and national authorities, as well as knowledge of particular gaps in protection which needed to be addressed. Another result was the increased cooperation and engagement between FRA and the Hellenic authorities, humanitarian organisations and EU deployed staff on the ground (Frontex and EASO), as well as an increase in duty bearers consulting FRA for informal advice (as before they were not available on the ground or present in meetings) and in FRA providing feedback to relevant duty bearers.

The research suggests that the key direct impacts of the FRA activities undertaken in the Greek and Italian hotspots had was raising the level of knowledge and awareness of the key fundamental rights issues by the relevant duty bearers at EU and national level, focussing the attention of these organisations on the most poignant issues identified on the

ground (e.g. protection of children). ‘ FRA have found a way to engage the In addition, the activities also had an internal side effect, authorities and have an impact, whereas namely the increase in FRA’s capacity to provide advice to the other actors have none. FRA has had a duty bearers on the ground (e.g. to develop tailor-made training significant impact in raising issues of a to deal with the practical challenges experienced on the human rights nature that could lead to ground), as a result of the increase in FRA’s knowledge of the an improvement of the current situation. situation on the ground and therefore the needs of the duty Some of the issues FRA raised have been bearers and rights holders. This again led to an increased level taken up when developing policy. For of authority of FRA’s interventions at Athens and HQ levels with example, FRA has been pushing a lot on

the Commission, Frontex and EASO. safe areas for unaccompanied minors in the hotspots, and the situation has since

All of the above had an indirect impact on policy development improved. ’

and practice on the ground in terms of fundamental rights European Commission official

protection in the hotspots, as the duty bearers including those officials directly dealing with the rights holders, such as the Greek authorities, EASO and Frontex, were incorporating the FRA feedback and recommendations, as well as knowledge gained, into their work.

The research also suggests that the most significant indirect impact of the Agency’s activities was in raising the profile of and the respect for fundamental rights with EU and national-level actors.

Impacts on duty bearers at EU and national level

In terms of the impacts of the activities undertaken by FRA in the hotspots on the duty bearers the main impacts can be summarised as:

 Raise awareness and remind duty bearers of fundamental rights more generally in the context of

the hotspots;

 Inform and work with the duty bearers on any specific fundamental rights concerns the FRA was

observing in the hotspots and;

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

 Follow up and ensure steps were undertaken to improve these issues raised by the FRA.

These activities ranged from distributing a regular confidential letter to the European Commission and Greek authorities pointing out FRA’s observations, to active engagement with duty bearers by participating in different (informal) meetings and discussions, by being available for any informal questions and ad-hoc advice, and finally by providing targeted training on specific fundamental rights issues.

Some stakeholders also noted the importance of the FRA’s presence and the FRA forming and vocalising its opinion on the human rights situation in the hotspots, especially to national and local actors; stakeholders noted that the FRA data was now the basic point of reference for all stakeholders in the field.

All these activities resulted in an increased awareness and knowledge of particular fundamental rights issues relevant in the context of the hotspots by relevant national authorities, EU Institutions and agencies. Moreover, it appears from the interviews undertaken as part of this case study that the fundamental rights issues raised by FRA were very relevant to the work of the duty bearers and were used by them when developing policies or guidance.

The FRA itself also noted that the area in which the FRA activity had most impact was ‘making the work of other actors [in the hotspots] more fundamental rights friendly’.

Other examples of situations where FRA’s suggested improvement materialised in concrete changes in policy or practice are for example:

 ‘ We did a lot internally with what the no use of handcuffs during return by sea (2016); FRA were suggesting. Whatever the FRA

 release of vulnerable people from Moria early April 2016; have recommended I have taken on

 board when developing policy, designing opening the gate connecting the residential area with the guideline or operations, such as the

service area in Vial in April 2016; guidelines for officers we sent to the

 the modalities of removals to Turkey (2016). hotspot. ’

Interviewee from Frontex

Moreover, several stakeholders noted that FRA training and workshops had an impact in changing practice. For example, one stakeholder noted in this regard that the training on return and return monitoring was the best training the FRA had ever provided and was extremely useful in their work.

Two stakeholders noted that the FRA activities had a limited impact in pushing legislation forward or on procedures. In addition, one stakeholder at the national level noted that FRA was currently having separate discussions with local national actors, but that they could further open up the dialogue by including a greater number of local and national authorities in each discussion forum in order to improve the coordination and create synergies.

Impacts on FRA internally

As summarised in the table above, the mapping and observations/visits of hotspots resulted in an increase in FRA’s knowledge of the situation on the ground and therefore an increased knowledge of the needs of the duty bearers and rights holders. This increased knowledge on the part of FRA has impacted in two ways.

Firstly, through the physical presence of the FRA in the hotspots and through the engagement of FRA with all the relevant national and EU-level stakeholders both in Athens and in the hotspots, FRA managed to get a reasonable picture of the fundamental rights challenges relating to the operation of

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

the hotspots. This additional knowledge strengthened FRA’s capacity to provide advice and tools to EU Institutions and agencies, specific and tailor-made to the practical challenges experienced in the Greek and Italian hotspots. However, the FRA interim assessment of its activities in Greece also acknowledged that given the continuously changing situation, the knowledge of FRA remained ‘patchy compared to EU

entities and humanitarian actors present on the ground’ which had substantially more staff. 246

Secondly, this first-hand knowledge of the situation on the ground increased the authority of FRA’s interventions at Athens and HQ levels with the Commission, Frontex and EASO. 247

Limitations to the impact

The impact of the activities undertaken by FRA in Greece and Italy seem to have been limited to some extent by the following limitations: 248

 Limited resources/staff: The project generally had very limited resources (e.g. 1 staff member and 1

support staff member in Greece in 2016), and as a result FRA had a minimal presence in Greece with an ability to deal only with a small set of issues. The Agency’s impact was curtailed by its limited resources, which forced the Agency to focus on a few major issues. If more resources had been diverted to the project, the Agency would have been able to undertake activities on a number

of issues in parallel.

 Lack of presence in Athens in 2016: FRA was not present in Athens in for a large part of 2016, which

significantly limited the possibility to follow up on findings and prevented progress on concluding the MoU. Moreover, the absence of the FRA in Athens delayed the establishing of a functioning

dialogue with the SRSS, which is the main EU communication channel with the Greek authorities.

 Lack of permanent presence in the hotspots throughout the project: Following the Agency’s first

deployment in Greece from April to September 2016, the lack of a permanent presence in the hotspots led to serious challenges in following up on the activities it had started. Due to the limited period of the Agency’s presence on the hotspots it was unable to develop the necessary informal channels that could improve the effectiveness of the Agency’s actions and the impact of its

activities.

 Limited visibility of the Agency: The impact of the Agency’s work in the hotspots was hampered by

the limited visibility of the Agency to national authorities, given its limited presence in Athens.

 Lack of an MoU with Greece until November 2016 249 : Because of the lack of an MoU for part of

2016, FRA regularly faced problems in accessing the hotspots or specific authorities.

Conclusion

‘Fundamental rights are a different language to other policy makers’

246 FRA, FRA’s expert advice at hotspots – Interim assessment and next steps, 4 August 2016.

247 FRA, FRA’s expert advice at hotspots – Interim assessment and next steps, 4 August 2016.

248 FRA, FRA’s expert advice at hotspots – Interim assessment and next steps, 4 August 2016.

249 The FRA and the Hellenic Ministry of Migration Policy signed a Memorandum of Understanding in November 2016 in order to facilitate and

deepen the cooperation between the Agency and the competent national authorities in Greece.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

The major unique contribution of the Agency was in providing fundamental rights expertise and training to EU and national authorities at the hotspots in Greece, and to an increasing extent in Italy. Stakeholders discussed the unique contribution of the Agency in improving procedures and guidelines for national and EU authorities regarding what information should be recorded and communicated to migrants. Their presence has been crucial in terms of providing fundamental rights advice in the EU’s

hotspots.

The Agency provided relevant advice to and undertook relevant activities for stakeholders on the ground. The training that the Agency provided was ‘very pertinent’ and well received targeted training. Through the development of tools, the Agency focused on issues under the thematic areas, such as vulnerable persons, rights of the child and unaccompanied minors. Furthermore, the Agency contributed significantly to the improvement of standard operating procedures in the hotspots,

particularly with regard to the thematic areas discussed above.

Regarding the efficiency of the Agency’s operations in the hotspots, there is considerable evidence that the resources available to the Agency limited the scope and impact of its activities. As discussed, the Agency’s resources were not sufficient in order to carry out this additional fundamental rights advice project without affecting the resourcing available for existing projects. Additionally, there is evidence, supported by interviewees, that the Agency’s actions within the hotspots were further limited due to a

strain on financial and human resources.

The Agency has engaged in effective collaboration with the Greek EU institutions and bodies on the ground (Commission, Frontex and EASO), and was considered significantly complementary to both national and EU level actors on the hotspots. The coordination with national authorities formed a much more challenging aspect of the work, as FRA previously had little visibility to these national authorities and no informal links and therefore was required to use formal channels, which entailed significantly more work and a reduced outcome. The bridging role the Agency played amongst the various EU and

national authorities formed an important aspect of its work in the hotspots.

Additionally, the Agency’s work in the hotspots helped to build first-hand knowledge of the situation on the ground within the Agency and increased the authority of FRA’s interventions with national and EU

authorities.

Regarding improvements to the Agency’s work on the hotspots, stakeholders discussed a need for greater follow-up on the implementation of the Agency’s recommendations at the EU and national level. Whilst the Agency had considerable impact in its training and advisory role, the Agency encountered significant challenges in ensuring the continuity of these improvements. Additionally, several fundamental rights gaps that the Agency highlighted were systemic deficiencies – such as the absence of a functioning guardianship system for unaccompanied children, which can only be addressed with considerable time and require policy change in Athens, which the Agency was unable to monitor and

ensure progress.

Furthermore, there remain significantly more tasks for the Agency to undertake at the hotspots. Additional activities that could increase the relevance of the Agency’s work include activities to bring all the local actors together, combined actions and synergies by the Agency and other actors, more public papers on hotspots and publicly available reports on best practices. Additional work on the rights of the people with disabilities, and on how to best identify vulnerable people, including victims of torture, sexual and gender-based violence, including trafficking in human beings, was discussed by stakeholders

as being relevant needs at the hotspots.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

In summary, whilst the Agency achieved considerable impact in its work on the hotspots, there is a need for the Agency to monitor and ensure the implementation of its recommendations by national and EU- level actors on the ground, which calls for a presence of FRA staff in Athens and in the hotspots on a

continuous basis.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

EU-MIDIS II – Muslim report social media plan

report launch analysis

The objective of the EU-MIDIS II – Muslims – selected findings report is to provide a unique insight into

experiences of discrimination of EU’s second largest religious group and to provide recommendations to

EU Member States to support integration and non-discrimination, as well as internal security policies.

Objectives of the social media plan:

As discussed in the social media plan of the report by the Agency, the objective of the posts on social media were:

 For rights holders to read more about the project on the FRA website

 For rights holders to read the results of the previous EU-MIDIS report in advance of the EU-MIDIS II

release

 To illustrate the cover page of the report to rights holders.

These objectives fall under a wider objective to raise awareness amongst the Agency’s stakeholders,

particularly rights holders, of the EU-MIDIS II – Muslim report and to assist in the dissemination of this report. Another sub-objective of the report itself is to raise the profile of the Agency with rights holders

regarding fundamental rights and raise awareness of the Agency’s other activities and outputs.

Awareness

As outlined in the social media plan, the Agency had planned to make nine posts on Twitter and

Facebook.

Regarding the number of page views and time spent on the publications page in the seven days following the release of the report, there was significant in traffic on the day of the release of the report

followed by a sharp reduction.

Figure 31: Page views and time spent on publication page during 7-day period after launch

When benchmarked against the downloads and page views during the first seven days after launch of a sample of the Agency’s other recent publications, the EU -MIDIS II – Muslim report received the second highest number of downloads, over 2,000, of the publication and the fourth highest number of page

views, over 3,500, as illustrated in Figure 32.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 32: Unique downloads and page views for FRA publications during the first 7 days after launch

When examining the number of downloads of FRA publications on the launch day, one day after the launch and one week after the launch, the EU-MIDIS II – Muslim report garnered significantly more downloads than any other FRA publication between 2015 and 2017, with almost twice as many

downloads on launch day as competing publications. However, the publications downloads in the day after launch and in the week following the launch are not significantly more than other Agency

publications, representing 28% and 26% above the second most downloaded publications one day after launch and a week after the launch respectively.

Table 24: Comparison of FRA publication downloads succeeding launch dates

Publication Launch day Launch day + 1 Downloads

downloads downloads during the

7 days after

launch

Second European Union Minorities and Discrimination 1,038 555 2,046 Survey. Muslims – Selected findings

Fundamental Rights Report 2017 175 126 520

Fundamental Rights Report 2016 521 381 1,327

Second European Union Minorities and Discrimination 140 246 895 Survey (EU-MIDIS II) Roma – Selected findings

Professionally speaking: challenges to achieving equality 376 274 1,373 for LGBT people

Handbook on European law relating to access to justice – 268 194 749 EN

Severe labour exploitation: workers moving within or into 705 433 1,622 the European Union

Surveillance by intelligence services: fundamental rights 661 367 1,492 safeguards and remedies in the EU

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

When comparing the unique downloads on launch day, launch day + 1, after 7 days and after 14 days, the significantly greater success in the dissemination and stakeholder engagement to the Violence Against Women (VAW) report against that of the EU MIDIS II – Muslim report becomes apparent. The ratio of downloads one week after launch of the VAW report is more than 4:1, which rises to almost 5:1 after two weeks after launch. However, comparison with the VAW survey must take into account the substantially larger geographic scope of the VAW, which covered all Member States in the EU, against

that of the EU-MIDIS II – Muslim report, which covered just 15 Member States.

Table 25: FRA publication downloads succeeding launch dates for select publications

Publication Launch day Launch day + 1 Downloads Downloads downloads downloads during the during the 7 days 14 days

after after launch launch

Second European Union Minorities and 1,038 555 2,046 2,307 Discrimination Survey. Muslims – Selected findings

Violence Against Women – Main results 2,206 2,637 8,408 10,685

Whilst the EU-MIDIS II – Muslim report is the top FRA publication download for September 2017, it is closely followed by the Violence Against Women report, which despite being released in 2014 still received over 2,000 unique downloads. This can be partially explained by the publications delayed released date, which ensured there was just 10 days in September for which the publication was publicly available. However, it is clear that the VAW survey continues to reach a significant audience and will act as a best practice in dissemination of Agency publications against which the EU-MIDIS II – Muslim report

should be measured.

Table 26: FRA publication downloads August–September 2017

Publication August 2017 September 2017 downloads downloads

EU-MIDIS II: Muslims – Selected findings NA 2,145

(the EU-MIDIS II – Muslim report was released on 21 September 2017)

Violence Against women – main results 1,679 2,101

Handbook on European law relating to asylum, borders and immigration IT 378 996

Handbook on European law relating to asylum, borders and immigration EN 754 917

EU-MIDIS II: Roma – Selected findings BG NA 780

Handbook on European non-discrimination law 428 630

Violence against women – At a glance ES 344 525

Handbook on European law relating to asylum, borders and immigration FR 165 518

The situation of Roma in 11 EU Member States 346 427

EU LGBT survey – European Union lesbian, gay, bisexual and transgender

survey – Results at a glance 339 406

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Similarly, annualised, the Agency’s publication with the highest number of downloads remains the Violence Against Women publication with over 20,000 downloads, considerably more than any other FRA publication, the closest of which is the Handbook on European law relating to asylum, borders and immigration at 8,803 downloads. Whilst it is impossible to extrapolate and estimate the annualised downloads of the EU-MIDIS II – Muslim report, should the report continue to be downloaded at a similar

level in the following 12 months, it is on course to be the second most downloaded FRA publication.

Table 27: Annualised 2017 FRA publication downloads

Publication Annualised 2017 downloads

Violence Against women – main results 20,896

Handbook on European law relating to asylum, borders and immigration EN 8,803

Handbook on European non-discrimination law 6,577

Handbook on European data protection law 6,155

Handbook on European law relating to asylum, borders and immigration IT 5,331

EU LGBT survey – European Union lesbian, gay, bisexual and transgender survey – Results

at a glance 4,148

Handbook on European law relating to asylum, borders and immigration FR 3,780

Violence Against women – At a glance PL 3,665

Legal capacity of persons with intellectual disabilities and persons with mental health

problems 3,626

Muslims in the European Union 3,602

Regarding the Agency’s webpages with the largest page views in September 2017, the EU-MIDIS II – Muslim report press release is the third most visited FRA webpage, with over 2,800 page views, significantly more than the webpage that contains the main results from the Violence Against Women

survey with 1,910 views in September 2017.

Table 28: FRA webpage views August–September 2017

Webpage August 2017 page September 2017 views page views

Home page EN 6,290 8,988

Vacancies list 6,046 7,385

EU-MIDIS II: Muslims – Press release NA 2,890

Vacancy – Senior Legal Officer 1,809 2,739

Vacancy – Operational Support Officer 1,930 2,258 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Webpage August 2017 page September 2017 views page views

About FRA 1,604 2,172

Recruitment 1,598 1,943

Violence Against women – main results 1,436 1,910

Internship page 1,440 1,847

Publications & resources home page 1,017 1,556

However, in a similar way as with the unique downloads, when comparing the dissemination and stakeholder engagement in page views of the VAW report and the EU-MIDIS II – Muslim report, the significantly greater success of the VAW report is further evidenced. The page views of the VAW report on launch day, launch day + 1 and one week after launch are all more than three times that of the EU- MIDIS II – Muslim report, with almost four times the page views two weeks after launch. The significantly higher page views on the launch day of the report highlights the considerable success of the stakeholder engagement in preparation for the release of the VAW report as well as the limited success

of the EU-MIDIS II – Muslim report in stakeholder engagement in comparison.

Publication Launch day Launch day + 1 Page views Page page views page views during the views 7 days during after the 14 launch days after launch

Second European Union Minorities and 1,689 872 3,477 3,477 Discrimination Survey. Muslims – Selected findings

Violence Against Women – Main results 5,230 2,809 12,074 14,282

Communication channels

With regard to the Agency’s selection of communication channels, it is important to understand where web traffic for the Agency’s website comes from and how users access these pages. The top referrer

websites to the Agency’s website are shown in Table 29.

Table 29: FRA top referrer webpages

Page August 2017 September 2017 Visits Visits

Facebook 345 1,322

Twitter 249 583 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Page August 2017 September 2017 Visits Visits

Trabalhar na União Europeia 46 109

EC.europa.eu 60 89

www.friatider.se NA 84

Similarly, Facebook and Twitter form the two largest referrer websites on the launch day of the EU- MIDIS II – Muslim report. Interestingly, four of the top seven referral websites are media organisations

from Germany, the Netherlands and Austria.

Table 30: FRA top referrer websites on 21 September 2017

Referrer website 21 September 2017 Visits

Facebook 184

Twitter 122

www.spiegel.de 45

www.spiegel.de 30

derstandard.at 23

www.dutchnews.nl 20

www.zeit.de 19

Regarding the Agency’s website, the EU-MIDIS II – Muslim report publication led to an increase in the website traffic after its release on 21 September.

Figure 28: FRA webpage traffic September 2017

However, once compared against the increase in the total website traffic following the release of the VAW report in 2014, the stakeholder engagement of the EU-MIDIS II – Muslim report, based on this

metric, was significantly lower than the VAW report. This further supports the evidence of a limited 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

engagement from stakeholders on the EU-MIDIS II – Muslim report, which was evidenced by the page views of the select publication pages. Therefore, the VAW report garnered significantly greater numbers

of page views of the publication page as well as total traffic to the FRA website.

Regarding the language that visitors used to access the FRA’s website and therefore the EU -MIDIS II – Muslim report, the visitors overwhelmingly browsed the website in English.

Figure 29: FRA webpage visitor language

Regarding search terms for visitors accessing the Agency’s website, the most common related key

search terms are ‘EU -MIDIS’ with six searches in September, which formed the 9th most popular term,

‘Muslims’ with six searches and ‘Muslim’ with a further four searches. When compared to the total

number of page views or downloads of the publication, the number of visitors who used search terms to find the publication forms less than 1% of the visitors.

When the number of views across communication channels are compared, direct entries, which includes emails, newsletters, and press releases on launch day, formed the key communication channel in

garnering views of the main publication page.

Figure 30: FRA referral webpages and number of referrals

2500 2150

2000

1500 1078 1104 Clicks from other sites on

1000 launch day 667

492 429

500 328298 442 121 Clicks from Facebook on

launch day

Direct entries (emails,email newsletters, press releases) on launch day

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Engagement

Table 31 illustrates the effectiveness of the social media posts of the Agency in the run-up to and on the launch day of the EU-MIDIS II – Muslim report, through the number of retweets, shares, likes, and views

of the social media posts by the Agency.

2 independent External Evaluation of the European Union Agency for Fundamental Rights nd

Final report

Table 31: Overview of the engagement of FRA social media posts for EU-MIDIS II – Muslim report

Date Planned Post Actual Post Channel Effectiveness

 Save the date! FRA findings on #humanrights experiences of  Save the date! FRA findings on #humanrights experiences of Muslims  33 likes

Muslims living in the #EU out 21 September #inclusiveEU living in the #EU out 21 September. #inclusiveEU (Multimedia post)

http://fra.europa.eu/en/project/2015/eu-midis-ii-european 2 ‘sad’

union-minorities-and-discrimination-survey  13 shares

11/09

 1 reply

 19 retweets

 19 likes

 In just one week FRA publishes findings on discrimination &  In just one week FRA will publish findings from #EU's largest dataset of  10 ‘likes’

harassment of Muslims living in #EU #inclusiveEU #humanrights experiences of Muslims living in the #EU, covering

http://fra.europa.eu/en/project/2015/eu-midis-ii-european#discrimination and harassment as well as sense of belonging and trust  2 ‘love’

union-minorities-and-discrimination-survey in democratic institutions. (Multimedia post)

14/09  21/09: FRA will publish findings from #EU’s largest dataset of  In just one week FRA publishes findings on #discrimination &  0 replies

human rights experiences of Muslims living in the #EU, harassment of Muslims living in #EU #inclusiveEU

covering #discrimination and harassment as well as sense of http://fra.europa.eu/en/project/2015/eu-midis-ii-european-union 18 retweets

belonging and trust in democratic institutions minorities-and-discrimination-survey (Multimedia post)

http://fra.europa.eu/en/project/2015/eu-midis-ii-european 15 likes

union-minorities-and-discrimination-survey #inclusiveEU

 FRA’s 2nd major survey on #humanrights experiences of  FRA's second major survey on #humanrights experiences of Muslims in  8 likes

Muslims in #EU out this week. 1st survey results: #EU will be published this Thursday. Find out more by taking a look at

http://fra.europa.eu/en/publication/2010/eu-midis-datathe results from the first survey. (Multimedia post)  2 shares

focus-report-2-muslims #inclusiveEU

18/09  FRA’s 2nd major survey on #humanrights experiences of Muslims in  1 reply

#EU out this week. 1st survey results

http://fra.europa.eu/en/publication/2010/eu-midis-data-focus-report 13 retweets

2-muslims #inclusiveEU (Multimedia post)

180 2 independent External Evaluation of the European Union Agency for Fundamental Rights nd

Final report

Date Planned Post Actual Post Channel Effectiveness

 10 likes

 On Action Day against Islamophobia and Religious Intolerance  On Council of Europe’s Action Day for countering Hate Speech against  20 likes

tomorrow, FRA will be publishing a report with findings from Muslims tomorrow, FRA will publish a report with findings from the

the largest dataset of comparable information on largest dataset of comparable information on #humanrights  2 shares

#humanrights experiences of Muslims living in the #EU. experiences of Muslims living in the #EU. #inclusiveEU #nohatespeech

#inclusiveEU (Multimedia post)  

20/09

 Tomorrow is Action Day against Islamophobia & Religious  FRA report on Muslims in the #EU out tomorrow, on the @coe Action  0 replies

Intolerance: a day to call for an end to #discrimination Day for countering Hate Speech against Muslims #nohatespeech

#inclusiveEU #inclusiveEU (Multimedia post)  54 retweets

 41 likes

 No planned post  Europe’s Muslims hampered by prejudice, warns study  0 replies

https://www.ft.com/content/cd3a76f6-9df4-11e7-8cd4-932067fbf946

20/09 … via @FT #inclusiveEU (Multimedia post)  16 retweets

 6 likes

 #EU Muslims have high trust in democratic institutions

despite experiencing widespread #discrimination http://fra.europa.eu/en/project/2015/eu-midis-ii-europeanunion-minorities-and-discrimination-survey #inclusiveEU

 The European Union Minorities and Discrimination Survey 21/09 6am (EU-MIDIS II): Muslims – selected findings is the second large

scale survey of Muslims carried out by the Fundamental Rights Agency. The survey asked about experiences of discrimination, harassment, police stops, and rights awareness, as well as markers of integration such as a sense of belonging and trust in public institutions.

181 2 independent External Evaluation of the European Union Agency for Fundamental Rights nd

Final report

Date Planned Post Actual Post Channel Effectiveness

 No planned post  @TimmermansEU: I want to assure Muslims in #EU that the European  36 replies

Commission will not tolerate intolerance. It is against our values &

21/09 laws.  16 retweets

 19 shares

 No planned post  Les musulmans sont bien intégrés en Europe  1 reply

http://www.lemonde.fr/international/article/2017/09/21/les

21/09 musulmans-sont-bien-integres-en-europe_5188855_3210.html …  5 retweets

#inclusiveEU (Multimedia post)  5 likes

 No planned post  Viele Muslime haben starke Bindung an ihr EU-Heimatland  4 retweets

http://www.zeit.de/news/2017-09/21/migration-viele-muslime

21/09 haben-starke-bindung-an-ihr-eu-heimatland 1 like

21060302?wt_zmc=sm.ext.zonaudev.twitter.ref.zeitde.share.link.x …

via @zeitonline #inclusiveEU

 If you’re a Muslim in the #EU, your name may be enough to  If you’re a Muslim in the #EU, your name may be enough to make sure  10 likes

make sure you never get a job interview. See you never get a job interview

fra.europa.eu/en/publication/2017/eumidis-ii-muslimshttp://fra.europa.eu/en/publication/2010/eu-midis-data-focus-report-  2 ‘laughs’

selected-findings #inclusiveEU 2-muslims #inclusiveEU (Multimedia post)

 ‘Our survey results make a mockery of the claim that Muslims

21/09 6:30am aren’t integrated into our societies. On the contrary, we see a  3 replies

trust in democratic institutions that is higher than much of

the general population,’ says FRA Director Michael O’Flaherty.  13 retweets

‘However, every incident of discrimination and hate crime

hampers their inclusion and reduces their chances of finding  3 likes

employment. We risk alienating individuals and their communities, with potentially perilous consequences.’

182 2 independent External Evaluation of the European Union Agency for Fundamental Rights nd

Final report

Date Planned Post Actual Post Channel Effectiveness

 No planned post  The vast majority of Muslims in the EU have a high sense of trust in  223 likes

democratic institutions despite experiencing widespread

discrimination and harassment, a major survey by FRA shows.  6 ‘sad’

http://fra.europa.eu/en/node/25179 (Multimedia post)  2 ‘loves’

21/09 7:30am  39 shares

 2 comments: 1 positive, 1

negative

 No planned post  On #InternationalDayofPeace 1 of our many star trainees uses FRA  40 likes

data to debunk myths about Muslims in the #EU. #jointogether. Read

more in the report: http://fra.europa.eu/en/node/25171 (Multimedia  5 ‘loves’

21/09 4pm post – video post) 

17 shares

 1,700 views

 Muslims in #EU face #discrimination when looking for work,  Muslims in #EU face #discrimination when looking for work, on the job  5 replies

on the job & when trying to access services & when trying to access services http://fra.europa.eu/en/press

22/09 9am http://fra.europa.eu/en/project/2015/eu-midis-ii-europeanrelease/2017/muslims-eu-high-levels-trust-despite-pervasive-  39 retweets

union-minorities-and-discrimination-survey #inclusiveEU discrimination … #inclusiveEU (Multimedia post)  15 likes

 FRA survey provides a unique insight into experiences of EU’s  FRA survey provides a unique insight into experiences of EU’s 2nd  10 replies

2nd largest religious group largest religious group http://fra.europa.eu/en/press

22/09 10am http://fra.europa.eu/en/project/2015/eu-midis-ii-europeanrelease/2017/muslims-eu-high-levels-trust-despite-pervasive-  31 retweets

union-minorities-and-discrimination-survey #inclusiveEU post) discrimination … #inclusiveEU (Multimedia  16 likes

 No planned post  EU survey finds Muslims willing to embrace non-Muslims  3 retweets

22/09 http://wapo.st/2xfQKAI?tid=ss_tw-bottom&utm_term=.69893c70b7a9

… #inclusiveEU  4 likes

183 2 independent External Evaluation of the European Union Agency for Fundamental Rights nd

Final report

Date Planned Post Actual Post Channel Effectiveness

 No planned post  The Muslim community, in all its diversity, has an important place in  6 retweets

the European landscape, says @TimmermansEU:

http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=604291  4 likes

22/09 …

 

 No planned post  EU-Bericht: Österreicher mit negativem Verhältnis zu Muslimen –  0 reactions

kurier.at https://kurier.at/politik/inland/eu-bericht-oesterreicher-mit

22/09 negativem-verhaeltnis-zu-muslimen/287.372.791 … via @KURIERat

#InclusiveEU (Multimedia post)

 Would Europe still be Europe if we were all the same? Great  On #InternationalDayofPeace 1 of our many star trainees uses FRA  2 replies

poem by FRA intern @Femi_Sorry on FRA's #discrimination data to debunk myths about Muslims in the #EU #inclusiveEU

22/09 survey #inclusiveEU #jointogether (Multimedia post - video)  33 retweets

 37 likes

 No planned post  The frequently repeated statement that Muslims won't or cannot  3 replies

integrate is nothing more than a myth @KAndruszFRA (Multimedia

22/09 post - video)  21 retweets

 14 likes

 No planned post  All information about FRA survey on #discrimination & #integration of  17 retweets

Muslims living in #EU incl methodology Q&A

22/09 http://fra.europa.eu/en/publication/2017/eumidis-ii-muslims 14 likes

selected-findings … (Multimedia post)

 No planned post  Two in five European Muslims have felt discriminated against – FRA  1 reply

survey results via @guardian #inclusiveEU (Multimedia post)

22/09  13 retweets

 4 likes

184 2 independent External Evaluation of the European Union Agency for Fundamental Rights nd

Final report

Date Planned Post Actual Post Channel Effectiveness

 No planned post  Full findings from FRA survey on #discrimination & #integration of  1 reply

minorities to be published 6/12 #inclusiveEU

22/09 http://fra.europa.eu/en/event/2017/reality-bites-experiences 10 retweets

immigrants-and-minorities-eu … (Multimedia post)  11 likes

185

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Engagement rates constitute the number of visitors that interact with the social media posts over the total number of people who view the post. When examining the Agency’s top tweets from 1 September until 28 September five of the top seven tweets concern the EU-MIDIS II – Muslim report, which garnered a total of 78,103 impressions in seven days. The tweets generated 1,227 engagements. The tweet that generated the highest engagement rate was the multi-media tweet that generated a similarly strong reaction response on Facebook, with over 1,700 views, as well as on Twitter. Whilst the engagement rate of the Agency’s tweets is significantly higher than the average across all commercial

industries of 0.05% recorded in 2017 250 and the average engagement rate of non-profits, as well as

being in line with the average engagement rate of non-profits dealings with human rights of 1.5%, 251 this

can still be considered a low engagement rate. The use of non-profits involved in human rights as a benchmark for engagement rates was due to the limited publicly available information on average engagement rates comparable across Facebook and Twitter as well as to the similar objectives of the Agency and non-profit organisations regarding the use of Twitter: to disseminate information and raise awareness amongst as large a group of its stakeholders as possible. Additionally, the choice of a human rights non-profit against which to benchmark the Agency’s engagement rate was made because it reduced the variation in engagement rates due to varying public interest in different thematic areas.

250 Rival IQ. 2017 Social Media Benchmarking report. Available at:

http://get.rivaliq.com/hubfs/eBooks/Rival_IQ_2017_Social_Media_Benchmark_Report.pdf?submissionGuid=44a7c3e3-4d45-4429-

92b2-b45af7d2af98

251 Rival IQ. 2016 Social Media Directors Guide to benchmarks.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Table 32: FRA top tweets 1–28 September

Regarding the Agency’s top Facebook posts from 1 September until 28 September, Table 33 illustrates that six of the Agency’s top 11 posts concerned or promoted the release of the EU-MIDIS II report. These six posts garnered a reach of over 21,200 visitors with an engagement of 1,306 across all the posts, which represents a total engagement rate of approximately 6%. Whilst this is slightly higher than the average engagement rate for a non-profit Facebook post of 5.4%, it is in line with the average engagement rate for non-profits dealing with human rights of 6.1%. 252 It is important to point out that 16,300 of the approximately 21,200 visitors were from just two of the six tweets concerning the EU- MIDIS II – Muslim report, highlighting the success of the multi-media post as well as the post announcing the report’s release on the day of publication.

252 http://www.mrss.com/lab/the-2016-social-media-directors-guide-to-benchmarks/

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Table 33: FRA top Facebook posts September 2017

In order to capture the response on Twitter to the Agency’s posts, we recorded tweets from the announcement date of the release of the report, 11 September, until 24 September. All tweets under the list of key search terms and hashtags were recorded. The search terms consisted of all hashtags used by the Agency in its social media posts as well as common search terms for the previous EU-MIDIS report and were used to ensure the inclusion of all relevant tweets by external stakeholders regarding

the EU-MIDIS II – Muslim report. The list of the 31 key search terms is shown in Table 34.

Table 34: Overview of key search terms used to track Twitter response to the EU-MIDIS II report release

Key search terms

 Human rights experiences of Muslims living  Bericht über die Diskriminierung von

in the #EU Muslimen in der EU

 Menschenrechtserfahrungen von  Rapport sur la discrimination à l'encontre des

Muslimen in der #EU musulmans dans l'UE

 Grundrechtserfahrungen von Muslimen in  #inclusiveEU

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Key search terms

der #EU

 Experience des droits fondamentaux des 

musulmans vivant dans L’UE InklusionEU

 European rights agency  inklusiveEU

 Europäische Menschenrechtsagentur  UE inclusif

 Europäische Grundrechtsagentur  #discrimination

 Agence des droits fondamentaux  #diskriminierung

 FRA findings  harassment of Muslims living in #EU

  FRA resultate Belästigung von Muslimen, die in der #EU

leben

 Résultats de FRA  Harcèlement des musulmans dans l’UE

 Minorities and discrimination survey  @EUrightsagency

 Minderheiten und

Diskriminierungserhebung

 Minderheiten und Diskriminierungsunfrage

 Sondage sur les minorités et les

discriminations

 Report on discrimination against Muslims

in the EU

Using these 31 key search terms for the time period 11 September to 24 September, 648 tweets were returned. Of these 648 tweets, 114 expressed a positive or neutral sentiment towards the release of the report. Neutral and positive sentiment tweets were grouped together due to the difficulty of distinguishing sentiment behind simple tweets which just include a link to the survey without expressing an opinion on the content of the survey. In contrast, 223 tweets expressed a negative sentiment towards the release of the publication. Within this group of negative tweets, there was significant range in the negativity of the sentiment and a substantial portion of tweets constituted anti-Muslim

sentiment. 311 of these tweets were unrelated to the release of the EU-MIDIS II – Muslim report.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Press clippings

Regarding the inclusion of references to the EU-MIDIS II – Muslim report in the press within the EU, the report was mentioned in 83 press clippings in the eight days following the release of the publication and

garnered significant media attention in major news organisations across the Member States.

Member States Number of press clippings reported by the Agency

AT 10

BE 3

BG 7

HR 0

CZ 1

DK 0

FI 1

FR 8

DE 14

EL 1

HU 1

IE 1

IT 5

LV 0

LT 0

LU 0

MT 1

NL 6

PL 3

PT 1

RO 0

SK 1

SI 2

ES 1

SE 1

UK 2 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Third countries

Algeria 2

India 1

Kuwait 1

Malaysia 1

USA 3

Turkey 3

As illustrated in the table above, the Agency recorded no publications in their press clippings from six Member States (RO, LU, LT, LV, DK, HR) and 38 of these 83 publications recorded were from just four Member States (AT, DE, FR, NL). The UK had just two press clippings recorded by the Agency but this is likely due to the criteria for the press clippings selection, as the survey results featured prominently in the Guardian and other major news publications. The survey results were discussed in news outlets in

six third countries, predominantly in countries where the majority of the population are Muslim.

Whilst the engagement and viewing figures for the traditional media reports discussing the EU-MIDIS II – Muslim report are impossible to quantify across the variety of news organisations, based upon readership figures of just a few of these major publications, the number of people who viewed survey results through the publication will be significantly higher than the viewing figures for the Agency’s social media posts regarding the publication. Additional investigation is required into the number of referrals to the EU-MIDIS II publication and download page, through both online and print articles in

Member States.

Conclusions

The Agency’s social media campaign to release the EU-MIDIS II – Muslim report was successful in generating visitors and reach and therefore in its objective to raise awareness of the report and its release. However, following the low engagement rate of tweets and the Agency’s social media plan’s potential overreliance on Twitter posts over Facebook and other social media platforms, as well as the success and higher engagement rate of the video multi-media post, it is recommended that the Agency make greater use of multi-media posts on Facebook with infographics, videos and other interactive elements. Additionally, there was little engagement in the run-up to the release of the report, due in part to the need to reschedule the launch date of the report at short notice, and the Agency could benefit from increasing the frequency of social media posts in the days preceding the publications release. Additionally, the tweet by Vice-President Timmermans was the post most commented on and retweeted by viewers, and further investigation is warranted into the possibility of including more posts by popular Commission figures, as well as the Agency’s Director. Another recommendation would be the

inclusion of hashtags specific to the EU-MIDIS II – Muslim report, that are not too broad as to capture

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

significant numbers of unrelated tweets such as the #inclusiveEU used, in order to more accurately record and analyse the sentiment of the tweets or posts under the publications-specific hashtag.

Further investigation is required into engagement rate of traditional print media as well as into developing media contacts to ensure that publications in all Member States follow the launch of a major publication. Additionally, the high ratio of tweets with negative sentiments in response to FRA and Vice President Timmermans tweets requires further investigation into potential mitigation measures, such as

joint posts alongside stakeholder organisations with large social media followings.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Appendix 1 – Full intervention logic

This section introduces the Agency, its mandate according to the Founding Regulation as well as the way it is organised, including planning and reporting mechanisms. It is presented in the form of an intervention logic, setting out:

 the needs the Agency was designed to address – one of the key elements of this study will be to identify whether the needs have evolved since the creation of the Agency in 2007 (including those

listed in the Impact Assessment for the creation of the Agency

253 ) and since the previous evolution

five years ago;

 the objective of the Agency – as set out in Article 2 of the Founding Regulation which has been operationalised in the five-year Strategic Plan (following thematic areas set out in the Multi-Annual Framework), and Annually in the Annual Work Programmes – there will consequently be different

levels of objectives assessed

 the inputs in terms of financial and human resources – in other words, the Agency’s budget and

staff structure;

 activities which refer to the operations of the Agency, its organisation and the tasks and projects it

undertakes.

 The effects of the Agency are then divided into outputs, results and impacts, which relate to the

different levels of objective and the needs.

Although the Commission’s ‘Roadmap’ on the Common Approach to EU decentralised agencies named the elaboration of common evaluation guidelines for EU agencies as an action to be undertaken 254 , no specific guidance, including within the 2015 Better Regulation Guidelines, has been provided as yet on the evaluation of EU Institutions, bodies and agencies.

253 Commission’s Proposal for a Council Regulation establishing a European Union Agency for Fundamental Rights {SEC(2005)849} /*

COM/2005/0280 final

254 European Commission (2012) Roadmap on the follow-up to the Common Approach on EU decentralised agencies, p. 8.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Commission evaluation guidelines (as part of the Better Regulation Guidelines) are considered to be generally applicable to Agencies inasmuch as the scope of the guidelines is concerned, which focuses largely on ‘retrospective’ evaluations. Better Regulation Guidelines have to be considered as the overarching framework within which the ‘Evaluation handbook for Agencies’, adopted by the EU Agencies network in 2014, should be situated. The handbook is a practical evaluation tool for EU decentralised Agencies, based on the combined experience of theoretical and practical methodologies for assessing performance, as used at present by the Agencies.

In addition, the external evaluation of FRA in 2012 did not make any recommendation regarding the creation of an evaluation framework for the Agency. However, the evaluation recommended that the Agency should review its priorities, refocus its activity around issues relevant to a number of Member States and provide a strategy for managing increasing demands of work, in order to respond to pertinent needs on fundamental rights issues within available resources.

With the above in mind, the intervention logic will be described using the terminology of the 2015

255

Better Regulation Guidelines and that specific to the Agency in its programmatic documents.

Table 35: Outlining our approach to the intervention logic of the European Union Agency for Fundamental Rights

European Union Agency for Fundamental Rights: Intervention Logic

The intervention logic is made up of the following building blocks, which are defined here and will be discussed throughout this chapter:

Needs – fundamental rights needs for anyone located in the European Union;

Objectives – set at the general, specific and operational levels, objectives detail the aims of the Agency, as defined based on the needs/problem;

Inputs – this term covers the financial and human resources available to the Agency in order to achieve the stated objectives;

Activities & Processes – the processes, tools and actions that are implemented by the Agency using the inputs available in order to achieve the intended objectives.

The spectrum of effects, consisting of:

Outputs – the direct products of the Agency’s activities; outputs are easily quantifiable and attributable to the Agency’s activities.

Results – the direct and immediate effects brought about by the Agency’s activities; results are less easily quantifiable and attributable to the Agency’s activities than outputs, but more so than impacts. Results are also specifically relevant to the direct stakeholders of the Agency’s activities.

255 European Commission (2015) Better Regulation Guidelines & Better Regulation ‘Toolbox’

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

European Union Agency for Fundamental Rights: Intervention Logic

Impacts – the effects of the Agency’s activities beyond the immediate effects (i.e. the impact on the wider economy/society); these can be direct or indirect, intended or unintended effects.

Needs

This section provides an overview of the fundamental rights needs/problems/issues in the EU for the period covered by the evaluation (2013 to 2017) and how these needs changed during this period.

In line with Article 2 of the Founding Regulation, this assessment will focus on the needs of the target groups of the FRA (i.e. the duty bearers), in terms of the type of assistance they need, and on which thematic areas. These duty bearers are:

EU Institutions, bodies, offices and agencies;

EU Member States (as well as candidate countries and potential candidate countries).

However, taking into account the spirit of the Founding Regulation, including references in the Regulation’s recital to Article 6 TEU, information on the needs of the rights holders has been added to this section. Although rights holders are not the direct target group of the FRA, they are the target group of the duty bearers to whom the Agency must provide assistance. Information on their needs will provide a deeper understanding of the needs the duty bearers are aiming to address and at the same time provide a better understanding of the context in which the FRA operates. Therefore, the needs of the rights holders are also relevant. It is important to note, however, that the FRA will not be evaluated against the impact it has on rights holders, but solely on the ways in which it addresses the needs of the duty bearers.

The data sources for understanding the needs of the FRA target groups was derived by undertaking desk research, as well as through an internal and external survey and interviews with:

The duty bearers themselves:

  EU level: officials of EU Institutions, bodies, offices and agencies; and

  Member States level: National Liaison Officers (NLOs); National Parliamentary Focal Points;

National Human Rights Institutions (NHRI).

The EU Fundamental Rights Agency (FRA);

The representatives of ‘rights holders’, such as international organisations responsible for protecting human rights (e.g. the Council of Europe – CoE and the United Nations High Commissioner for Refugees – UNHCR) and civil society and NGOs.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Objectives

The general objective of the FRA is set out in Article 2 of the Founding Regulation. The Multi-Annual Framework 2013–2017 (MAF) sets out the areas where the FRA general objective (set out in Art 2 of the Regulation) is to be achieved through Tasks (set out in Art. 4 of the Regulation) implemented on the basis of the strategic priorities and thematic objectives defined by the Management Board in the

256

Strategic Plan. The Strategic Plan 2013–2017 contains different levels of objectives, at the thematic and strategic level, including expected impacts, expected results, planned activities and actions. In addition, the Mid-Term Review of the Strategic Plan 2013–2017 introduced revised specific objectives and expected results, as well as the main operational focus until the end of 2017 for each strategic

priority and thematic objective. Finally, every year Annual Work Programmes (AWP) were adopted,

As part of the effort to align the Strategic Plans and the Annual Work Programmes, from 2016 all EU agencies had to draft a Single Programming Document in which both components had to be included. In 2016, the first Single Programming Document was drafted for FRA for the period 2017–2019, to ensure a more consistent approach between annual and multi-annual programming and define the common elements of the work programme (providing a common terminology for objectives, activities, outputs, results, impacts, etc.). The Single Programming Document reports on the achievements made in the period 2013–2015, as well as objectives, expected results and priorities for each specific priority and for each thematic objective. Moreover, the document explains the link between strategic and thematic objectives with the Performance Measurement Framework (PMF). The second Single Programming

Document will be drafted in 2017.

Based on the Agency’s specificities, these different levels of objectives have been translated into:

 General objective set out in the recitals and Article 2 of the FRA Founding Regulation;

 Thematic objectives: as set out in the Strategic Plan for 2013–2017;

 Strategic priorities and objectives: as set out in the Strategic Plan for 2013–2017;

 Operational objectives: No specific operational objectives were set in the Regulation, Multi-Annual

Framework or Strategic Plan. The Strategic Plan 2013–2017 does provide some level of operational objectives (e.g. ‘key actions’ for each strategic objective and ‘key actions’ for each thematic objective). Moreover, the Annual Work Programmes (AWP) list the proposed projects for a particular year under each thematic area. Therefore, for the purpose of this study the operational objectives relate to the projects of the FRA. Given that the Agency has adopted activity-based monitoring and budgeting, the objectives set out in the Performance Measurement Framework will

be used to assess the projects on their effectiveness.

Based on the interviews undertaken with FRA staff and further analysis of the programming

256 FRA Strategic Plan 2013–2017, http://fra.europa.eu/sites/default/files/fra_strategic_plan_en.pdf

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

documents and the FRA operating model in particular, another level of objectives (‘thematic objectives’) has been added to the intervention logic.

These different levels of objectives are presented in greater detail in the sections below.

General objectives

The objective of the Agency is laid down in Article 2 of its Founding Regulation (see below). The FRA founding Regulation enshrines the principles of liberty, democracy, respect for human rights and

fundamental freedoms and the rule of law. 257

‘The objective of the Agency shall be to provide the relevant institutions, bodies, offices and agencies of the Community and its Member States

Council Regulation (EC) when implementing Community law with assistance and expertise relating

N° 168/2007 Art.2 to fundamental rights in order to support them when they take measures

or formulate courses of action within their respective spheres of

competence to fully respect fundamental rights.’ 258

Article 2 thus requires FRA to assist the EU Institutions and agencies, and Member States in implementing EU Law. In this regard, recital 3 emphasises that the EU and its Member States must respect fundamental rights when implementing EU law. Recital 9 further states that ‘the Agency should refer in its work to fundamental rights within the meaning of Article 6(2) of the Treaty on European Union, including the European Convention on Human Rights and Fundamental Freedoms, and as reflected in particular in the Charter of Fundamental Rights, bearing in mind its status and the accompanying explanations. The close connection to the Charter should be reflected in the name of the Agency.’ Therefore, it can be concluded that the general objective of the FRA is to assist the EU and its Member States to respect fundamental rights as enshrined in the Charter of Fundamental Rights of the European Union. The Charter is the first point of reference for the FRA when offering its evidence-based advice on how to ensure respect for fundamental rights. 259 In December 2009, with the entry into force

of the Lisbon Treaty, the Charter was given binding legal effect equal to the Treaties.

In order to achieve this objective, the Agency performs the following main tasks, as stated in Article 4 of the Founding Regulation.

257 http://www.eu2006.at/en/Austria/International_Organisations/EUMC.html

258 Council Regulation (EC) No 168/2007 i of 15 February 2007 establishing a European Union Agency for Fundamental Rights

259 http://fra.europa.eu/sites/default/files/fra-2014-booklet_en_0.pdf

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Tasks of Agency – article 4 of the Regulation

• a. 

  collect, record, analyse and disseminate relevant, objective, reliable and comparable information and data, including results from research and monitoring communicated to it by Member States, Union institutions as well as bodies, offices and agencies of the Community and the Union, research centres, national bodies, non-governmental organisations, third countries and international organisations and in particular by the competent bodies of the Council of Europe;

• b. 

  develop methods and standards to improve the comparability, objectivity and reliability of data at European level, in cooperation with the Commission and the Member States;

• c. 

  carry out, cooperate with or encourage scientific research and surveys, preparatory studies and feasibility studies, including, where appropriate and compatible with its priorities and its annual work programme, at the request of the European Parliament, the Council or the Commission;

• d. 

  formulate and publish conclusions and opinions on specific thematic topics, for the Union institutions and the Member States when implementing Community law, either on its own initiative or at the request of the European Parliament, the Council or the Commission;

• e. 

  publish an annual report on fundamental-rights issues covered by the areas of the Agency’s activity, also highlighting examples of good practice;

• f. 

  publish thematic reports based on its analysis, research and surveys;

• g. 

  publish an annual report on its activities; and

• h. 

  develop a communication strategy and promote dialogue with civil society, in order to raise public awareness of fundamental rights and actively disseminate information about its work.

Pursuant to Article 3 of the Founding Regulation, FRA is only allowed to carry out the tasks listed above

within the thematic areas laid down in the Multi-Annual Framework (MAF), 260 a document which is

adopted by the Council. For 2013–2017, there were nine thematic areas, namely:

• 1. 

  access to justice;

• 2. 

  victims of crime, including compensation to victims;

• 3. 

  information society and, in particular, respect for private life and protection of personal data;

• 4. 

  Roma integration;

• 5. 

  judicial cooperation, except in criminal matters;

260 Council Decision No 252/2013/EU establishing a Multi-AnnualFramework for 2013-2017 for the European Union Agency for Fundamental

Rights.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

• 6. 

  rights of the child;

• 7. 

  discrimination based on sex, race, colour, ethnic or social origin, genetic features, language, religion or belief, political or any other opinion, membership of a national minority, property, birth, disability, age or sexual orientation;

• 8. 

  immigration and integration of migrants, visa and border control and asylum; and

  • 9. 

    racism, xenophobia and related intolerance.

It should be noted that FRA is allowed to go beyond these thematic areas, e.g. criminal matters, when carrying out scientific research and surveys, preparatory studies and feasibility studies, as well as when it is developing conclusions and opinions on specific thematic topics, but only when requested by the European Parliament, the Council or the Commission. 261 Moreover, these conclusions, opinions and reports may concern proposals from the Commission under Article 293 of the Treaty or positions taken by the institutions in the course of legislative procedures, as long as they do not deal with the legality of acts (as per Article 263 of the Treaty) or with the question of whether a Member State has failed to fulfil

an obligation under the Treaty. 262

Thematic objectives

As a result of the first independent external evaluation of the Agency, 263 and bearing in mind the Common Approach to the decentralised agencies and the subsequent guidelines introduced by the EU Institutions and the Performance Development Network, the Management Board built upon the recommendations to define the FRA’s Strategic plan, which ‘outlines the main strategic priorities and thematic objectives, set within the Agency’s Multiannual framework for the following five years’. The FRA Strategic Plan 2013–2017 includes thematic objectives which align to a large extent with the thematic areas of the Multi-Annual Framework (MAF) for 2013–2017. The only difference is that in the Strategic Plan ‘access to justice’ has been combined with ‘judicial cooperation’ and one thematic objective: ‘FRA Surveys – horizontal fiche’ has been added. Under each of these thematic areas a main

thematic objective is listed (see Table 36).

Table 36: Thematic objectives, as set out in the Strategic Plan 2013–2017

Thematic objective area (as

per the Strategic Plan) Main thematic objective (as per Strategic Plan)

Access to justice including to contribute to the EU’s efforts to enhance mechanisms for ensuring judicial cooperation access to justice through judicial as well as non-judicial mechanisms, at the EU and Member State level, which serve to underpin

261 Article 5 (3) Regulation in conjunction with Article 4(1)(c) and 4(1)(d) Regulation.

262 Article 4(2) Regulation

263 External Evaluation of the European Union Agency for Fundamental Rights, Ramboll, November 2012.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Thematic objective area (as

per the Strategic Plan) Main thematic objective (as per Strategic Plan)

fundamental rights compliance

Victims of crime, including to become a main actor in the promotion of the fundamental rights of compensation to victims of crime victims; in particular as regards the right of victims to have crime access to justice, and to make visible the invisible victims of, in

particular, hate crime

Information society, in to establish FRA as a relevant player in the field of information society, particular, respect for private privacy and data protection life and protection of personal data

Roma integration to contribute to the achievement of the goals set by the EU Framework on Roma integration to respect the fundamental rights of Roma across the EU in the context of the implementation of the EU 2020 Strategy.

Rights of the child to mainstream child rights issue throughout the Agency’s work, in line with the need for a holistic approach, and thus provide robust, relevant and timely evidence-based advice on the protection, respect and promotion of the rights of the child to EU Institutions and Member States

Discrimination 264 to provide robust, policy relevant and timely evidence-based advice

on equality and non-discrimination fundamental rights challenges to EU Institutions and Member States as well as other stakeholders

Immigration and integration of to promote effective fundamental rights safeguards in the migrants, visa and border development and implementation of EU policies in the field of control and asylum Immigration and integration of migrants, visa and border control and

asylum

Racism, xenophobia and to serve as an observatory/data warehouse on phenomena of racism, related intolerance xenophobia and related intolerance including hate crime and

extremism, as well as on racial and ethnic discrimination

FRA Surveys – horizontal fiche to ensure FRA becomes a key actor providing robust and policyrelevant comparable data for use as evidence-based advice on

264 Full thematic area name: Discrimination based on sex, race, colour, ethnic or social origin, genetic features, language, religion or belief,

political or any other opinion, membership of a national minority, property, birth, disability, age or sexual orientation 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Thematic objective area (as

per the Strategic Plan) Main thematic objective (as per Strategic Plan)

fundamental rights at the EU and Member State level.

Source: Strategic Plan 2013–17

As can be seen in Table 37, each thematic objective in the Strategic Plan is accompanied by a set of specific objectives, key activities and expected results. Moreover, the Annual Work Programmes (AWP) sets out annual objectives for each of the nine thematic objectives listed in the Strategic Plan. Finally, the Annual Activity Reports do not report achievements by the nine thematic objectives, but for each of

the thematic areas listed in the Multi-Annual Framework (MAF).

Table 37: How the FRA documents report on the thematic objectives

Strategic / Planning Document Implementation document

Multi-annual Annual Multi-annual Annual

FRA document FRA Strategic Plan Annual Work Mid-Term Review Annual Activity 2013–2017 (SP) Programmes (AWP) of the FRA Report

Strategic Plan 2013–2017

Content FRA Sets nine thematic Sets annual Reports on Reports on document ‘main objectives’, objectives for each achievements achievements by

(objectives it and for each thematic objective made between thematic area of sets and objective: of the SP: 2013–2015 MAF (not thematic

effects it  ‘specific  ‘expected objective)

reports on) objectives’ outcomes’

 ‘key activities’  ‘planned

activities’

 ‘expected

results’  ‘planned

outputs’

In 2016 progress made towards the achievements of the thematic (and strategic) priorities in the Strategic Plan was assessed in the Mid-Term Review of the FRA Strategic Plan 2013–2017. Although the Mid-Term Review ‘largely reaffirmed the existing strategy’, the Mid-Term Review set the revised objectives and expected results, as well as what the main operational focus of the FRA should be in 2016–2017 for each thematic area, in order to fulfil completely the set objectives by the end of 2017 and ‘to respond to the context changes and emerging needs for assistance and expertise in FRA’s areas

of work’.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

Strategic Objectives

265

In addition to the thematic objectives, the Strategic Plan for 2013–2017 lays down the Agency’s Strategic Objectives. They cover a range of activities and goals that the FRA is aiming to achieve over the defined period. As shown below, the strategic priorities are more process oriented and relate to how the Agency is to better achieve its general objective and thematic objectives (i.e. the means to an end /

the mechanism to achieve the other objectives).

Strategic Priorities 2013-17

 Strategic Priority 1: Enhancing FRA’s contribution the processes at EU level

 Strategic Priority 2: Enhancing FRA’s contributions to processes at national level

 Strategic Priority 3: Identifying trends over time and measuring progress in Member States

 Strategic Priority 4: Developing timely and targeted responses to fundamental rights

emergencies

 Strategic Priority 5: Improving the impact of FRA communication and awareness raising

 Strategic Priority 6: Planning FRA work and evaluating its impact

The Strategic Plan also provides some context around each strategic priority, as well as one or more ‘main objectives’, expected results and planned key actions for each priority. Table 38 sets out an

example for Strategic Priority 1.

Table 38: Different levels of objectives for each strategic Priority

Strategic Priority Main strategic objectives Expected results Key actions

1 Enhancing FRA’s contribution  FRA enhances its  FRA is  A database of

to processes at EU level relevance for legal and consulted all FRA policy processes at EU where opinions is level fundamental created

 FRA responds to rights concerns 

requests for opinions arise

Human and financial

and advice in a timely  Findings and resources for

and competent analysis providing

265 FRA Strategic Plan 2013-2017, http://fra.europa.eu/sites/default/files/fra_strategic_plan_en.pdf

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Strategic Priority Main strategic objectives Expected results Key actions

manner produced by advice

 FRA enhances its FRA are easily functions are coordination with the available allocated in European Commission,  Data and FRA work

the European expertise programmes

Parliament and the provided by  Contacts with

Council of the FRA are other EU actors

European Union increasingly are increased

 FRA cooperates taken up by EU

efficiently with other Institutions

EU agencies and civil society

Moreover, the previously mentioned Mid-Term Review sets the revised objectives and expected results, as well as what the main operational focus of the FRA should be in 2016–2017 for each

strategic priority, in order to fulfil completely the set objectives by the end of 2017 and ‘to respond to the context changes and emerging needs for assistance and expertise in FRA’s areas of work’.

As can be seen in Table 39, the Annual Work Programmes for 2012, 2013 and 2014 do not mention the strategic priorities and therefore do not include any further annual objectives under each Strategic Priority (as is the case for the thematic objectives). However, since 2015 the Annual Work Programme includes a specific section on the FRA Strategic priorities for 2013–2017.

The Annual Activity Reports do report on the achievements under each strategic priority.

Table 39: How the FRA documents report on the thematic documents and strategic objectives

Strategic / Planning Document Implementation document

Multi-annual Annual Multi-annual Annual

FRA document FRA Strategic Annual Work Mid-Term Review Annual Activity Report Plan 2013– Programme of the FRA

2017 (SP) 2015 and 2016 Strategic Plan 2013–2017

Content FRA Sets six Reports on Reports on Reports on document strategic progress for achievements achievements by

(objectives it sets priorities, and each strategic made between strategic priorities and and effects it for each priority 2013-2015 objectives

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Strategic / Planning Document Implementation document

Multi-annual Annual Multi-annual Annual

reports on) priority:

 ‘main

objective s’

 ‘expected

results’

 ‘key

actions’

Operational Objectives

As stated above, no specific operational objectives were set in the Regulation, MAF or Strategic Plan. The Strategic Plan 2013–2017 does provide some level of operational objectives (e.g. ‘key actions’ or

planned activities’); however, these are related to the thematic or strategic objectives in particular.

Under this intervention logic, the operational objectives relate to the projects and outputs of other activities of the FRA. These ‘non-project activities’ can relate to any activity not planned in the same way as projects and include opinions and on-the-ground activities (such as the presence of FRA staff on Greek islands during the refugee crisis, etc.). The Annual Work Programme (AWP) lists the proposed

projects for a particular year under each thematic area.

One of the key challenges of the reconstruction of the intervention logic is to reconcile the Agency’s existing evaluation and monitoring framework, the Performance Measurement Framework (PMF), with

the Agency’s strategic and operational objectives set out above.

As stated above, the Single Programming Document shows how these strategic and thematic objectives relate to the PMF (see Figure 36).

Figure 36: Overview of the links among the Programming Document, Performance Measurement Framework and Target Repository

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Source: FRA Single Programming Document 2017-2019, Annex VIII – Performance Measurement Framework, 2016.

The Single Programming Document also links:

 each of the strategic ‘main objectives’ under each Strategic priority (except strategic priority 6) with

a corresponding PMF objective;

 each of the thematic ‘specific objectives’ under each thematic objective (except the thematic

objective on ‘FRA Surveys – horizontal fiche’) with a corresponding PMF objective.

However, the Single Programming Document does not cross reference the Thematic objectives with the Strategic objectives.

The operational objectives of the Agency relate to its (intended) outputs and are referred to as ‘objectives related to output’ in the Performance Measurement Framework (PMF). This forms a causal link to ‘objectives related to short-term impact’, ‘objectives related to long-term impact’ and eventually ‘aspirational impacts’ in the Performance Measurement Framework (PMF). However, operational objectives, and the related indicators, can be used to build a bottom-up approach to assessing the extent to which the Agency achieves its strategic and thematic objectives. The table below provides an initial iteration of the way in which the ‘objectives related to output’ and the related indicators can contribute to evidencing impacts relating to the strategic objectives. A similar approach can be taken for the Thematic objectives; however, most ‘objectives related to outputs’ would be relevant for most

thematic areas, as most of these activities have been undertaken across all thematic areas.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

PMF Objectives MAF Strategic Objectives

Objectives Objectives Objectives Strategic Priority Strategic Priority Strategic Priority Strategic Priority Strategic Priority Strategic Priority related to related to shortrelated to 1: Enhancing 2: Enhancing 3: Identifying 4: Developing 5: Improving the 6: Planning FRA long-term term impact outputs FRA’s FRA’s trends over time timely and impact of FRA work and

impact contribution the contributions to and measuring targeted communication evaluating its processes at EU processes at progress in responses to and awareness impact

level National level Member States Fundamental raising Rights

emergencies

O1: Provide

assistance and expertise to EU Institutions and

LT1: FRA’s ST1 :

Produce Member States assistance and

expertise and/ or

disseminate O2: Formulate provided to

support legal effective Opinions and and policy evidence-based other advice to EU developments recommendations Institutions and related to and conclusions Member States

Fundamental from research

rights (EU level) findings, Opinion O3: Develop

and other policy research analysis advice to decisionand surveys and makers and key identify good stakeholders. practices

LT2: FRA’s O4: Publish

assistance and thematic reports expertise and handbooks

provided to

support legal ST2: Improve O5: Disseminate and policy effective nettargeted

206 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

PMF Objectives MAF Strategic Objectives

Objectives Objectives Objectives Strategic Priority Strategic Priority Strategic Priority Strategic Priority Strategic Priority Strategic Priority related to related to shortrelated to 1: Enhancing 2: Enhancing 3: Identifying 4: Developing 5: Improving the 6: Planning FRA long-term term impact outputs FRA’s FRA’s trends over time timely and impact of FRA work and

impact contribution the contributions to and measuring targeted communication evaluating its processes at EU processes at progress in responses to and awareness impact

level National level Member States Fundamental raising Rights

emergencies

developments working and information to the related to development of right stakeholders

Fundamental synergies among and at the right rights (Member EU and national time

State level) stakeholders, and

Civil Society O6: Develop

relevant methods and standards

O7: Develop

effective networks with key partners and Agency networks

ST3: Increased awareness of O8: Develop

fundamental awareness-raising rights among materials stakeholders

207 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Effects (Outputs; Results; Impacts)

In this intervention logic, the effects of the Agency (outputs and results) mirror the different levels of objectives developed in section 2.2.3. The final level of effect, impacts, reflect the ways in which the needs identified have been addressed, both for the target groups to which the Agency provides its assistance (i.e. the EU Institutions, bodies, offices and agencies and the EU Member States) and beyond

(and potentially indirectly) to rights holders in the European Union.

Linking to the Performance Measurement Framework of FRA and in line with its Performance Indicator system the effects of the Agency’s activities can be broken down into four levels:

 outputs

 short-term impacts

 long-term impacts

 aspirational impacts.

FRA’s activity spans across several thematic areas as defined by its Multi-Annual Framework, and includes a large number of projects. In line with the Agency’s Performance and Evaluation frameworks, during the planning phase each project is given a set of outcome objectives as well as short-term, longterm and aspirational indicators that it needs to fulfil. After the completion of the project, the measurement indicators assist the Agency in assessing the effects of the project on its output requirements but also on how it promotes the entire organisation’s short-term, long-term and

aspirational objectives.

Table 40: Example of project with allocated output indicators and short-term, long-term and aspirational indicators 266

Project Output indicators Short-term, long-term, aspirational indicators

Biometric data in large IT  Number of hearings or  Number of references to FRA’s findings

borders, immigration and presentations to and opinions in EU and Member State asylum databases – institutional policies, reports, and legislative

fundamental rights stakeholders across preparatory work

concerns levels of governance  Number of references by civil society

 Number of research actors to FRA findings

activities [per type of 

task, per thematic area, Stakeholders that consider that per type of outputs, per reliable, relevant and high-quality

geographical area] information resulting from FRA data collection, research and analysis

 Number of relevant activities is delivered to the intended

stakeholders receiving target group

FRA information  Stakeholders who agree that FRA

conclusions and recommendations

266 Annual Work Programme 2016, FRA, December 2015

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Project Output indicators Short-term, long-term, aspirational indicators

inspire or influence EU policy and legislative changes

Outputs

Outputs are the first level of effects; they are defined as the direct products of the FRA’s activities. Outputs are the most easily identifiable and quantifiable effects and are also the easiest to attribute to

the intervention – they relate to the Operational Objectives of the Agency.

The outputs of specific projects can be found in the MATRIX information system. In order to illustrate the outputs indicator measurement for FRA’s projects and in an effort to reconstruct the intervention logic of the Agency we hereby present the outputs of the FRA project: Fundamental rights safeguards and remedies in light of surveillance by national authorities. The output measurements for the project

include, inter alia:

 Publication of report on surveillance by national intelligence services examining fundamental rights

safeguards and remedies in the 28 Member States of the EU;

 The preliminary findings for the report were presented to the European Parliament and

consequently the Parliament published a resolution addressing the need to safeguard citizens’

fundamental rights;

 The Council of Europe’s Committee of Ministers referred to the Agency’s activities in this area in a

reply to the Parliamentary Assembly Recommendation on mass surveillance;

 The agency’s findings were provided to and discussed with the Council of Europe Venice

Commission in the context of its work on democratic oversight of the security services;

 The report was launched at the World Forum for Democracy in Strasbourg;

 FRA met with representatives of the EU Institutions, the Council of Europe and other bodies to

discuss further activities related to the topics of the report;

 FRA presented the report’s findings at an official hearing of the Italian parliamentary committee for

the security of the Republic;

 FRA participated in an inter-parliamentary committee meeting on democratic oversight of national

intelligence services at the European Parliament;

 Other demonstrable outcomes and outputs (such as (social) media coverage etc.).

These achievements are used to assess the project’s success and contribute to the aggregated effects of

the thematic area that the project belongs to.

The Annual Activity Reports include a selection of output indicators usually aggregated at a thematic area level. Although the number and nature of the indicators that FRA uses to monitor its activity vary

annually depending on the tasks undertaken, of particular interest are the FRA core indicators that

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

remain the same throughout the evaluation period. Table 41 presents a selection of output indicators as extracted by the Agency’s Annual Activity Reports for the years 2013, 2014 and 2015 267 for each thematic area – this only provides an overview per thematic area; numerous outputs for all the three

objectives listed are produced in the cross-cutting activities.

Table 41: Example of output indicators grouped in thematic areas for the years 2013, 2014 and 2015

Output objective 4: Output objective 1: Output objective 7:

Publish thematic Provide assistance and Develop effective

reports and expertise to EU networks with key

handbooks Institutions and partners and agency Member States network

Thematic areas Number of Number of hearings or

publications presentations to Number of

produced (per institutional networking events

thematic area) stakeholders across organised levels of governance

2013 2014 2015 2013 2014 2015 2013 2014 2015 Access to justice 2 5 1 - 16 48 - 2 66

Cross-cutting activities 6 6 4 - 5 7 - 50 3

Discrimination 11 4 4 - 5 27 - 8 13

Immigration and asylum 4 6 9 - 2 83 - 8 15

Information society - 6 2 - 5 14 - 3 4

Racism 6 1 4 - 8 15 - 3 2

Roma integration - 4 - - 3 17 - 0 7

Rights of the child - 1 6 - 0 25 - 2 5

Victims of crime - - 2 - 47 4 - - -

It is important to note here that for this analysis only the published indicators that were similar across FRA’s Activity reports have been included. The number of publications, hearings or networking events organised by MAF area will naturally change from year to year depending on the particular stage that any research project is at – meaning that in some years a project will be at the publication and

dissemination stage, while in other years there may be a focus on data collection.

Results

Results are the second level of effects; they are defined as the direct and immediate effects brought about by the FRA’s activities. Often, the results relate to the effect of an activity on the target group or

stakeholders involved. Results relate to the Thematic Objectives developed in section 2.2.3.

In line with FRA’s Performance Measurement Framework, the Results is a broad category of impacts that include the short-term and long-term impacts. The same aggregation processes are followed in measuring these effects: projects may be assigned a number of short-term and long-term objectives

that contribute to the overall thematic area effect. Furthermore, FRA’s thematic areas and their

267 The Annual Activity Report for the year 2016 has not been published yet. The Annual Activity Reports are usually published at the start of the

following year – in this case in 2017.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

priorities aggregate to a higher level developing the strategic priorities that FRA has established in its Strategic Plan for 2013-2017.

A summary of the results achieved in a given year can be found in FRA’s Annual Activity Reports. A specific example of these effects is presented below, in the context of FRA’s strategic priority: Enhancing FRA’s contribution to processes at EU level. In 2015, the Agency, in regard to its cooperation

with the EU Institutions, inter alia:

 Contributed to, and participated in, 19 meetings – including hearings – at the European Parliament.

 Engaged with the European Parliament’s Committee on Employment and Social Affairs (EMPL), to

which it presented its report on severe forms of labour exploitation in the EU.

 Engaged with the Committee on Petitions (PETI), to which it presented its work on rights of people

with disabilities.

 Presented its Annual report 2014 on fundamental rights challenges and achievements to the

Council’s Working Party on Fundamental Rights, Citizens’ Rights and Free Movement of Persons (FREMP) and to the European Parliament’s Committee for Civil Liberties, Justice and Home Affairs (LIBE) – the main findings were reflected in the Justice and Home Affairs Council conclusions on the

application of the Charter of Fundamental Rights in 2014.

 In 2015, the new system to evaluate EU Member States’ implementation of the Schengen acquis

became operational. FRA supported the European Commission through training, information sharing and expert advice during on-site evaluation missions.

 The Commission’s call for proposals for action grants to support transnational projects to combat

sexual harassment and sexual violence against women and girls included FRA’s survey results as evidence supporting taking this action.

These achievements show the effect of the Agency’s work beyond the operational outputs of its projects and activities as they directly involve its stakeholders and network.

The long-term effects of the Agency’s activity are particularly important in assessing its overall effect. A more in-depth investigation in the Agency’s short-term and long-term indicators as achieved by its

projects must be conducted in the context of the proposed evaluation.

Impacts

Impacts are the third and highest level of effects; they are defined as the effects of the Agency’s activities beyond the immediate effects. However, impacts are the most difficult effects to quantify and identify; and it is also difficult to attribute any effect to a specific activity due to the influence of external factors. Furthermore, impacts are closely related to the general objectives outlined above and,

beyond this, to the needs the Agency is designed to address.

The evaluation will investigate any high-level changes in the area of fundamental rights and identify the links to the Agency’s activities. These high-level changes may take the form of general trends over a

long period of time.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Appendix 2 – Survey results

See attached spreadsheets

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Appendix 3 – List of stakeholders consulted

Interviews

Last name First name Position Organisation

BEIS Michail Head of Sector FRANET and Annual Report, ECR

BURELLO Luca Head of Sector Quality, CSD

CATALA Xavier Head of Sector Finance and Budget, CSD

FIKATAS Nikolaos Head of Sector ICT and Facilities, CSD

GAJDOSOVA Jana Head of Sector Access to Justice, FJ

IVANOV Andrey Head of Sector Roma and Migrant Integration, ECR

IYIGUENGOER Aydan Head of Sector Awareness Raising, FRPD

KARAKITSOU Eleni Head of Sector Human Resources, HRL

LARIMIAN Cathrin Head of Section Media & Communication Innovation, FRPD

LIGUORI Saverio Head of Sector Planning, CSD

NEVALA Sami Head of Sector Statistics and Surveys, FJ

NICKELS Henri Head of Sector Equality, ECR

PODSIADLOW

SKI Astrid Head of Sector Rights of the Child, ECR

ROMAIN Nicole Head of Sector Editing and Production, ECR

SILVESTRI Adriano Head of Sector Asylum, Migration and Borders, FJ

VAN DER

VEUR Dennis Head of Sector Consultation & Cooperation, FRPD

Accardo Andreas Head of Director´s Office, DIR

Dimitrakopoul Head of Equality and Citizens' Rights Department,

os Ioannis ECR

GOODEY

Joanna Head of Freedoms and Justice Department, FJ

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

MANOLOPOU Constantino

LOS s Head of Administration, CSD

O'FLAHERTY Michael Director

ROSCAM

ABBING Friso

Head of Fundamental Rights Promotion Department, FRPD

TOGGENBUR

G Gabriel Senior Legal Advisor, DIR

Shams Asadi Human Rights Coordinator City of Vienna

Benedek Wolfgang ETC Graz, FRANET partner Universitaet Graz

Maltese President’s Farrugia Ruth Director General Foundation for

the Wellbeing of Society

Field Ophelia Consultant

Follmar-Otto Petra Project Manager German Institute for Human Rights

European Gaspard Anne Director network of

Equality Bodies

Gasperlin Marco MB Chair Frontex, Senior Police Superintendent Slovenian Police

Hatzi Chrysi Greek Ombudsmen

Hoefmans Alexander Head of Unit of Fundamental Rights of the EU Belgium/Ministry of Justice

Huotarinen Heini Unit for Democracy, Language Affairs and Finland / Ministry Fundamental Rights of Justice

Karimian Michael Human Rights Program Manager Micrsoft

Kemper Jutta Former German Deputy NLO

Koppel Olari Director, Office of the Chancellor of Justice

Kuuse Rait Deputy Secretary General on Social Policy, Ministry

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

of Social Affairs

Ireland / Irish

Logan Emily Head of Office Human Rights and Equality

Commission

Amnesty McGowen Iverna Amnesty International Brussels International

Brussels

Morijn John JHA Counsellor Dutch permanent representation, Ministry of FRA contact point during the Dutch presidency Interior

Department of Justice and

Deaglan Principal Equality, Diversity

Briaian and Equality Law

Division

Quá Eduardo High Commission for Migrations

Raba Kristi Permanent representation of Estonia to the EU, Head of section Justice

Rautio Sirpa Director, Finnish Human Rights Centre

Rudolf Beate Director German Institute for Human Rights

President of the Rurka Anna President of the CoE’s Conference on INGOs CoE’s Conference

on INGOs

Sandrisser Wilhelm Ministry of the Interior

Sofianidou Athina Expert Greece / Ministry of Interior

Suurpää Johanna Director, Unit for democracy and Fundamental Rights, Ministry of Justice ADD

Yatropoulos Adviser on European and International issues / France /

Mantzari Nepheli Equinet contact point Defender of Rights

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Adamo Chiara DG JUST

Arnaez Inmaculada Fundamental Rights Officer Frontex

Carreira Josef Director EASO

Chistenshen Dorthe Sec Gen

European Corrado Laura DG Home, Head of Unit Migration Commission

Officials

Covassi Beatrice Cappo della Rappresentnza in Italia, European Commission

HADJIYIANNIS

, Ioannis in Athens

Structural Reform Support Services

Kaessner Ralph General Secretariat, Fundamental Rights and Criminal Justice, DG D – Justice and Home Affairs Council of the EU

Körner Bernd Deputy Director Frontex Frontex

Lindholm Pia Team Leader, DG-JUST – [NON- DISCRIMINATION/ANNUAL REPORT] DG JUST

Mihaylova Milena Sec Gen

European Parliament / Committee on Civil Liberties,

Mlinar Angelika Justice and Home Affairs / Rapporteur on European

Multiannual Framework Parliament

Nagy Josef European Parliament / MEP/ EP Rapporteur of European annual resolution on fundamental rights Parliament

NALEWAJKO Pawel General Secretariat, Fundamental Rights and Criminal Justice, DG D – Justice and Home Affairs Council of the EU

European Nielsen Henrik DG Home, Head of Unit Asylum Commission

Officials

Ravo Linda Policy Officer – Fundamental Rights – [HATE CRIME] DG JUST

Rueda Bueso Amparo European Parliament / Committee on Civil Liberties, European Justice and Home Affairs/ Secretariat Parliament

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Saastamoinen Salla Director for Civil Justice, DG JUST DG JUST

Serpieri Massimo Deputy Head of Unit D1, Non-discrimination Policies and Roma Coordination – [ROMA] DG JUST

Setti Valeria DG JUST

Stevens Helga MEP / Coordinator of main political groups in LIBE European Committee Parliament

Stockwell Nathalie DG JUST

Tuite Margaret , Unit C1, Fundamental rights and Rights of the Child – [RIGHTS OF THE CHILD] DG JUST

Zdechovski Thomas MEP / Member of the LIBE Committee European Parliament

Colzi Barbara Field Office Chios (Greece) UNHCR

UN Office of the

D’Achamps Paul Acting Regional Representative for OHCHR, EU High Commissioner for

Human Rights

Denstad Finn Senior Sector Officer – Inter-Institutional Coordinator Norway Grants

Gallach Cristina United Nations Under-Secretary-General for Communications and Public Information

Khorozyan Therese UNHCR

Link Michael Director of the OSCE Office for Democratic Georg Institutions and Human Rights OSCE

Valeriu Council of Europe Nicolae

Directorate

Giakoumopou General Human

los Christos Director Righs and Rule of Law – Council of

Europe

Di Robilant Filippo Ex MB IT FRA management Board

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Entzinger Prof. Han Chairperson FRA Scientific Committee

Erdmane Anete Management Board member, Latvia FRA Executive Board

Seidensticker Frauke Lisa Chairperson, Management Board FRA Executive Board

Geddes Phiip Management Board member Management Board

FRA Executive

Hofmann Rainer Management Board member, Council of Europe Board / FRA Management

Board

Vice-Chairperson, FRA Executive

Nowak Manfred Co-Director of the Ludwig Boltzmann Institute of Human Rights (BIM). University of Vienna Board / FRA

Management Board

Rojas Coppari Pablo FRP Advisory Panel Migrant Rights Centre Ireland

Ryngbeck Annica FRP Advisory Panel

President of the HRH Board / AP

Sarnavka Sanja FRP Advisory Panel member 2014 – 2016. Human

Rights House Zagreb

Centro d'Iniziativa Scagliotti Luciano FRP Advisory Panel per l'Europa del

Piemonte

Schulze Marianne FRP Advisory Panel Austrian League of Human Rights

Tuulikki

Rautio Sirpa Member of the MB

FRA Management Board

BRILLAT Regis Contact person at the Department of the European Social Charter and Social Security Code Council of Europe

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Gorris Ellen DG JUST, Unit C1, Fundamental rights and Rights of the Child – [RIGHTS OF THE CHILD] DG JUST

HEIREBAUDT-

DANLOS Marie-Laure DG BUDG

In’t Veld Sophie European Parliament / MEP/ EP Rapporteur for EP European report on a mechanism for the Rule of Law Parliament

Kinga Gal European Parliament / Committee on Civil Liberties, European Justice and Home Affairs Parliament

Leggeri Fabrice Director Frontex

Meyer Kathrin Executive Secretary to International Holocaust Remembrance Alliance IHRA

NOGUEIRA Fernando

European Parliament / Committee on Civil Liberties,

Post Soraya Justice and Home Affairs / Shadow Rapporteur on European

Multi-AnnualFramework Parliament

Roger Christine Director for Home Affairs in the General Secretariat of the Council of the European Union Council of the EU

Ruete Matthias Director-General, DG HOME DG HOME

European Schmidt Szabolcs DG Justice, Non-discrimination Commission

Officials

European

Seiffarth Oliver DG Home, Unit C2, Border Management and Schengen Commission

Officials

Sippel Birgit European Parliament / Committee on Civil Liberties, European Justice and Home Affairs Parliament

Human Rights and Tserdsvadze Tinatin Human Rights and Democracy Network Democracy

Network

Värk Kristi Head of Department, EU and Foreign Relations Department, Ministry of the Interior

Vedsted

Hansen Jens Member of the MB

Professor of law, Aarhus University

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Von

Bogdandy Armin Former Scientific Committee Member

FRA Scientific Committee

Wikstroem Cecilia European Parliament / Committee on Civil Liberties, European Justice and Home Affairs Parliament

Director, Human Rights and Integration Directorate,

Agius Silvan Ministry for Social Dialogue, Consumer Affairs and University of

Civil Liberties Malta

Director, Human Rights and Integration Directorate,

Agius Silvan Ministry for Social Dialogue, Consumer Affairs and University of

Civil Liberties Malta

Artjoki Sari Head of Representation, European commission Representation in Finland

Czech Republic / Baršová Andrea Director Human Rights and Minority Protection Office of the

Government

Boček Tomáš Special Representative of the Secretary General on Migration and Refugees Council of Europe

Boldrini Laura President of the Chamber of deputies

Boni Michal MEP / Member of the LIBE Committee European Parliament

Brandvik Tove FRP Advisory Panel

Buttigieg Victoria NLO

Charlier Patrick Former MB Member, Chair of the AREDIT MB FRA Management Working group Board

Chinnici Caterina European Parliament / Committee on Civil Liberties, European Justice and Home Affairs Parliament

Cras Steven General Secretariat, DG D 2B Judicial Cooperation in Criminal Matters

Doujak Gerhard Deputy NLO Austria

Draghici Damian /Committee on Civil Liberties, Justice and Home European Affairs Parliament

DUDZICZ Klaudia DG BUDG 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Fajon Tanja MEP / Member of the LIBE Committee European Parliament

Feeney Martina Director, Human Rights Unit, Department of Foreign Affairs ADD

European Parliament / Committee on Civil Liberties,

Ferrara Laura Justice and Home Affairs / EP Rapporteur on European

Fundamental Rights Parliament

France / National

Forst Michel Head of Office Consultative Commission on

Human Rights

Friggieri David DG-JUST, Coordinator anti-Muslim Hatred – Fundamental Rights – [RACISM] DG JUST

Galea Vincent Clerk to the House of representatives

Gerritsen Jan DG BUDG

Greece / Ministry

Giantsiou Katerina Special Secretary Roma inclusion of Labour, Social Insurance and

Social Solidarity

Gonzales Pons Esteban Vice Chair of the EPP, responsible for legal and home European affairs Parliament

Vicepresident.

Gonzalez Valentin FRP Advisory Panel Movement Against

Intolerance

Grabbe Heather Open Society Foundation Open Society Foundation

Gramatikov Martin Scientific Committee member Scientific Committee

Grech Dr Peter Attorney General

Consigliere Politico

Gualdi Francesco Segreteria Particolare On. Sandro Gozi Sottosegretario di Stato alla Presidenza del Consiglio

dei Ministri per le Politiche e gli Affari europei 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Hakonen Kimmo General Secretary, Office of the Chancellor of Justice

HARTWIG Marc-Arno sits in Rome DG Home

European

Heimans Dick DG JUSTI/B1 – general criminal law; Deputy head of Unit Commission

Officials

Hennessey Mary-Ann COE

Husz Dora DG JUST, Unit D1 Non-discrimination Policies and Roma Coordination – [ROMA] DG JUST

Hynes Cormac Department of Foreign Affairs

Open Society Israel Butler Former Policy analyst at Foundation

Brussels

Jääskeläinen Petri Parliamentary Ombudsman

JAEGER Markus Contact person at the CoE/ FRA activities Council of Europe

Jöeorg

Jurtšenko Mariko

Adviser, Legislative Policy Department, Ministry of Justice

European

Kohner Debbie Director Network of National Human

Rights Institutions

Korhonen Pasi Director, International Affairs, Ministry of Social Affairs

Lambeck Tobias DG BUDG

Leicht Lotte Human Rights Watch Human Rights Watch

Long Gina Clerk to the Justice and equality Committee, House of the Oireachtas (Irish Parliament )

Lunacek Ulrike MEP / Vice-president of the European Parliament European Parliament

Lykovardi Kalliopi Senior Investigator, Equinet contact point Greece/ Office of

Greek 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Ombudsman

Greece / Centre For European Constitutional

Mousmouti Maria Project Manager – Executive Director, Director of International Projects Law –

Themistokles and Dimitris Tsatsos Foundation

European Asylum Martini Ermina Senior Officer Civil Society Support Office

(EASO)

Marttunen Matti Committee Counsel Finland National Parliament

Massa Maryanne Private Secretary to the President ADD

Mc Bride Jeremy Former Chair of the Scientific Committee FRA Scientific Committee

Michael O’Boyle Former Deputy Registrar of the European Court of European Court Human Rights of Human Rights

Milašiūté Vyganté Lithuanian NLO

FRA Executive

Mirosław Wróblewski Management Board member, Poland Board / FRA Management

Board

Moraes Claude Chair of the LIBE Committee European Parliament

Moschos George Greece / Ombudsman

European Münz Rainer European Political Strategy Centre (EPSC) Commission

Officials

FRA Executive Board / FRA

Nemitz Paul Management Board member, European Commission Management Board / European

Commission Officials

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Niedermueller Peter European Parliament / LIBE Committee European Parliament

Orbán Szilvia Hungarian NLO

Pakosta Liisa Commissioner, Gender Equality and Equal Treatment

Paradis Evelyne ILGA Europe ILGA Europe

Pardalis Panos FRP Advisory Panel

Pastrana Eva Council of Europe ‘HELP in the 28’ Programme Coordinator

National Institute Simon Patrick Director of Research for Demographic

Studies (INED)

Pimiä Kirsi Non-Discrimination Ombudsman ADD

Poliart Anne

European Council Pollet Kris Senior Legal and Policy Officer on Refugees and

Exiles (ECRE)

Ricard-Nihoul Gaëtane Deputy Head of Representation, European Commission Representation in France

Sakslin Maija Former Chair of MB FRA Management Board

Schölin Anna Karin Swedish NLO

Siitam-Nyiri Kristel Deputy Secretary General on Criminal Policy, Ministry of Justice

Spinelli Barbara European Parliament / Committee on Civil Liberties, European Justice and Home Affairs Parliament

Spring Caroline Department of Justice

Stavropoulou Maria Director Greek Asylum Service Greece / Asylum Service

Sule Satish DG JUST 2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Last name First name Position Organisation

Pfanner Susanne Austrian National Contact Point for Roma Integration Austria / Federal Chancellery

Tsedev Hulan Chief of the department at HQ for Europe UNHCR

Cyprus / Office of

Tsiartas Aristos Head of Anti-discrimination Body the Commissioner of Administration

and Human Rights

Van Den Unit C1, Fundamental rights and Rights of the Child –

Bogaert Sina [RIGHTS OF THE CHILD] DG JUST

Segretario del Comitato Interministeriale per gli

Verola Nicola Affari Europei Capo della Segreteria del Sottosegreterio agli Affari

Europei

European

Vidovic Lora Chair Network of National Human

Rights Institutions

Von Team Leader – Policy coordinator for dialogue with

Schnurbein Katharina churches – [ANTISEMITISM] DG JUST

Wouters Jan Professor Institute for International Law

Executive

Wroblewski Miroslaw ex Executive Board member and ex Management Board member board/manageme

nt board

Bulgaria / Yotova Kristina Second Secretary Ministry of

Foreign Affairs

List of stakeholders surveyed

See attached

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights Final report

FRA Impact – Twitter Sentiment analysis

Introduction

In order to fully investigate the Agency’s reach and impact, a sentiment analysis assessing the views of select Agency outputs and events on Twitter was required. This was done through a process of computationally identifying and categorising opinions expressed in a piece of text, especially in order to determine whether the writer's attitude towards a particular topic, product, etc. is positive, negative, or

neutral 268 .

This approach categorised these tweets (positive/negative/neutral) in order to provide an indication of stakeholder attitude to select FRA outputs and activities. This analysis focussed on the messages tweeted in the thirty days surrounding the Fundamental Rights Forum in 2016 and the release of the Fundamental Rights Report in 2017, identified as significant outputs and events by the Agency during

the 2013-2017 programming period.

Defining Sentiment

For the purposes of our research, we define sentiment to be ‘a personal positive or negative feeling’. An example of which has been provided below:

Table 42: Sentiment analysis key terms definition

Sentiment Query Article/Content

Positive FRA ‘The FRA is providing very relevant input.’

Neutral FRA ‘The FRA is a EU decentralised Agency.’

‘The FRA has not helped the Negative FRA fundamental rights situation in

the EU.’

Procedure

Data Collection

The data set for the analysis was purchased by the research team through a third-party organisation. A test of the data was done manually, which produced a proof of concept of the sentiment attribution to

a high confidence level. Interactive tools were used to automatically extract the content from the

268 Lexycal analytics: https://www.lexalytics.com/technology/sentiment

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

tweets to assist in the classification task and an interactive application was built for the classification analysis.

Table 43: Time period of data sets used in Twitter sentiment analysis by output and activity

Activity/Output Publication date/ Date of Recorded tweet dataset activity

Fundamental Rights Forum 2016 20-23/06/2016 15/06/2017 – 15/07/2017

Fundamental Rights Report 30/05/2017 30/05/2017 – 30/06/2017 2017

The specific datasets used for the analysis were selected due to their coverage of tweets, under the key relevant search terms outlined below, by stakeholders both immediately preceding and proceeding the Fundamental Rights Forum in 2016, providing a sentiment analysis of over 1,000 initial responses tweeted under the key search terms on the event on Twitter as well as providing an analysis of over 2,000 tweets under the key search terms in the 30 days following the release of the Fundamental Rights

Report in 2017.

The key search terms and the time series of the data set were the key determinants of the size of the used to capture tweets forming the datasets for the sentiment analysis. The Agency’s assigned Twitter handles for the Fundamental Rights Forum and the Fundamental Rights Report in English, German and French were included to capture the tweets in response to Agency posts, similarly variations of these handles that captured the greatest number of tweets were included, as well as the Agency’s own Twitter handle to maximise the number of potential relevant tweets. There was a positive but nonlinear relationship between the inclusion of a greater number of search terms and an increase in the size of the dataset as the increase in the number of search terms led to a reduction in the percentage of relevant tweets within the newly captured tweets especially with regard to those captured under the Agency’s own Twitter handle. Therefore, following significant investigation of these parameters, the datasets chosen for the sentiment analysis were selected to ensure the highest number of relevant tweets by Agency stakeholders on the Fundamental Rights Forum in 2016 and the Fundamental Rights

Report in 2017.

by Agency stakeholders between the five days preceding the fundamental rights forum in 2016 and of the stakeholder response to the release of the fundamental rights report output released in 2017. In order to maximise the size of the potential dataset of tweets relevant to these outputs and activities, the following search terms in English, German and French were used. Whilst including the number of languages the key search terms were translated into did increase the number of potential tweets captured, the increase was moderate and the sentiment analysis was limited by the availability of dictionary of weighted positive/negative words that performed with a high degree of confidence only in

English, French and German.

Table 44: Key search terms used in Twitter sentiment analysis by output and activity

Activity/Output Twitter Key Search Terms

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Activity/Output Twitter Key Search Terms

 ‘#rightsforum16’  ‘fundamental rights forum’  ‘Grundrechteforum’ OR ‘Grundrechteforum’

Fundamental Rights Forum 

2016 ‘forum sur les droits fondamentaux’

 ‘forumsurlesdroitsfondamentaux’  ‘#EUrightsagency’  ‘@EUrightsagency’  ‘#rightsreport17’  ‘fundamental rights report’  ‘Grundrechte-Bericht’

Fundamental Rights Report  ‘Grundrechtebericht’

2017  ‘rapport sur les droits fondamentaux’

 ‘rapportsurlesdroitsfondamentaux’

 ‘#EUrightsagency’

 ‘@EUrightsagency’

Methodology

Once the messages were extracted from dataset, the task for each file was to:

 Call Aylien API to analyse the content (http://aylien.com/);

 If the API does not return any analysed text, it tries to extract the article’s content using Boilerpipe

(https://github.com/rasmusjp/boilerpipe.net) and then analyses it with Aylien Api;

 If it still cannot find the content it looks for redirects using HtmlAgilityPack and the redirected page

is analysed by Aylien Api.

The attached sentiment analysis data report contains:

 An excel report with the tweets analysed content, language, polarity, polarity confidence,

subjectivity and subjectivity confidence;

Results – Fundamental Rights Forum 2016

1,202 tweets from between 15 th th June 2016 and 15 July 2016 were successfully categorised through

the sentiment analysis. From these 1,202 articles whose content was analysed for its sentiment, 13%, or 151 tweets, were categorised as having a positive sentiment when discussing the Fundamental

Rights Forum 2016. In contrast, 19%, or 227 tweets, were considered to have a negative sentiment regarding the Forum. However, the vast majority, 69% constituting 824 tweets, in response to the

Fundamental Rights Forum in 2016, under the key search terms were categorised as having a neutral sentiment.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 31: FRA Twitter sentiment analysis polarity of reactions to the Fundamental Rights Forum 2016

19% 13%

69%

Positive Neutral Negative

Table 45 illustrates the confidence level of the sentiment categorisation, with an average confidence in the categorisation of sentiment of the tweets of 75.5% and a mode of 71%. However, significant range exists between the minimum and maximum confidence of sentiment attribution of approximately 66%

due substantially to a small number of outliers.

Table 45: Confidence of sentiment attribution for tweets surrounding the Fundamental Rights Forum 2016

Function Score

Mean 0.775186775

Median 0.791830778

Mode 0.71036756

Min 0.344215095

Max 0.999917269

Range 0.655702174

Regarding the subjectivity of the articles, the table below illustrates the fact that just 24% of the captured tweets that mention the Forum under the key search terms are subjective, explaining the

significant number of neutral sentiment tweets identified.

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Figure 32: FRA Twitter sentiment analysis subjectivity analysis of reactions to the Fundamental Rights Forum 2016

24%

76%

Subjective Objective

There is a high level of confidence in the subjectivity categorisation with a mean of 99.4%, a median and mode of 100%. However, there is a similarly significant range in the confidence levels of the

subjectivity categorisation across the tweets, which is due to a small number of outliers.

Table 46: Confidence of subjectivity attribution for tweets surrounding the Fundamental Rights Forum 2016

Function Score

Mean 0.994474996

Median 1

Mode 1

Min 0.519359587

Max 1

Range 0.480640413

Conclusion

Following the sentiment analysis, we can, with a high degree of confidence, state that the majority of tweets collected by the Agency under the key search terms used in the analysis, a majority of which are

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

objective tweets, display a neutral sentiment towards the Agency. Similarly, we can conclude that whilst this is followed by the significant number of tweets, approximately 19%, that display a negative sentiment towards the Fundamental Rights Forum 2016, that the proportion of tweets that displayed a positive sentiment towards the Forum, with a high degree of confidence, is, in comparison, not

significantly lower, at 13%.

Results – Fundamental Rights Report 2017

2,297 tweets on the Fundamental Rights Report 2017 captured under the key search terms were successfully categorised through the sentiment analysis. From these 2,297 tweets, whose content was analysed for its sentiment, 15% or 338 tweets were categorised as having a positive sentiment when discussing Fundamental Rights Forum 2017. This contrasts favourably to the 6%, or 144 tweets, that were considered to have a negative sentiment regarding the report. Similar to the Fundamental Rights Forum, the significant majority of the captured tweets, 79% or 1815 tweets, were categorised as having

a neutral sentiment of the Fundamental Rights Report in 2017.

Figure 33: FRA Twitter sentiment analysis polarity of reactions to the Fundamental Rights Report 2017

6%

15%

79%

Positive Neutral Negative

The table below illustrates the confidence level of the sentiment categorisation, with an average confidence in the categorisation of sentiment of the tweets of 75.5% and a mode of 71%. However, a similarly significant range exists between the minimum and maximum confidence of sentiment

attribution of approximately 60.6% due substantially to a small number of outliers. .

Table 47: Confidence of sentiment attribution for tweets surrounding the Fundamental Rights Report 2017

Function Score

Mean 0.779449176

Median 0.782221943

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Function Score

Mode 0.71036756

Min 0.392393529

Max 0.998327076

Range 0.605933547

Regarding the subjectivity of the articles, the table below illustrates the fact that over 90% of the

tweets surrounding the fundamental rights report 2017 are objective.

Figure 40: FRA Twitter sentiment analysis subjectivity analysis of reactions to the Fundamental Rights Report 2017

10%

90%

Objective Subjective

There is a high level of confidence in the subjectivity categorisation with a mean of 99.5%, a median and mode of 100%. However, there is a similarly significant range in the confidence levels of the

subjectivity categorisation across the tweets of over 47%, which is due to a small number of outliers.

Table 48: Confidence of subjectivity attribution for tweets surrounding the Fundamental Rights Report 2017

Function Score

Mean 0.994910538

Median 1

Mode 1

2 nd independent External Evaluation of the European Union Agency for Fundamental Rights

Final report

Function Score

Min 0.526716591

Max 1

Range 0.473283409

Conclusion

Following the sentiment analysis, we can, with a high degree of confidence, state that the majority of tweets in 30 days following the report’s release that were captured under the key search terms, a

substantial majority of which are objective tweets, displayed a similarly neutral sentiment towards the Agency’s Fundamental Rights Report 2017 as against the Fundamental Rights Forum in 2016. In contrast to the 13% of tweets captured in response to the Fundamental Rights Forum 2016 that

expressed a positive sentiment of the event, this report was followed by a comparable percentage of tweets expressing, with a high degree of confidence, positive sentiment, constituting 15% of the tweets

captured under the key search terms, this, in fact represented significantly more nominal tweets due to the increased dataset of captured tweets. Additionally, this analysis highlights the relatively small number of tweets that displayed a negative sentiment, with a high degree of confidence, regarding the

Fundamental Rights Report 2017.

London | Brussels | Los Angeles | New York | Washington, DC | Zurich

Optimity Advisors, Ltd www.optimityadvisors.com www.twitter.com/optimityeurope