Legal provisions of COM(2015)479 - Final evaluation of the Eurostars Joint Programme (2008-2013)

Please note

This page contains a limited version of this dossier in the EU Monitor.

dossier COM(2015)479 - Final evaluation of the Eurostars Joint Programme (2008-2013).
document COM(2015)479 EN
date September 30, 2015
EUROPEAN COMMISSION

Brussels, 30.9.2015

COM(2015) 479 final

REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL

Final evaluation of the Eurostars Joint Programme (2008-2013)


1. Introduction

In July 2008, the European Parliament and the Council adopted a decision providing for EU participation in financing the Eurostars Joint Programme (‘Eurostars’) 1 . The Eurostars Decision limited the EU financial contribution to ‘the equivalent of a maximum of one third of the effective contributions of the participating Member States and the other participating countries, 2 within a ceiling of EUR 100 million.’ The participation was for the duration of the Seventh Framework Programme for Research (2008-2013). Since then, the European Parliament and the Council have adopted a new Decision providing for continued participation in funding in the new Eurostars-2 joint programme from 2014 to 2020 (Eurostars-2 Decision). 3  

Eurostars aims to provide financial support to transnational market-oriented research projects initiated and led by R&D-performing 4 small and medium-sized enterprises (SMEs). 5 These firms should be able to carry out the greater part of a project’s R&D work, and should be able to exploit its results commercially, thus improving their competitive position. Eurostars projects are collaborative, meaning that in any project, there should be at least two partners (autonomous legal entities) from two different participating states, with at least one being an R&D-performing SME. Projects must have a maximum duration of three years, and within two years of project completion, the product of the research should be ready for launch onto the market.

The EUREKA 6 Secretariat, based in Brussels, acts as the central support unit for the network and is the dedicated implementation structure for the programme. The Secretariat organises calls for proposals, verifies the eligibility of applications and selects projects for funding. It is also responsible for allocating the EU financial contribution. National funding bodies in the participating countries earmark in their R&D budgets the national contributions to Eurostars and finance their national participants. This funding mechanism is called a ‘virtual common pot’. In summary, Eurostars operates on the basis of centralised evaluation, but decentralised funding.

The Eurostars Decision required an interim evaluation to be carried out by the Commission two years after the start of the programme and for the results to be communicated to the European Parliament and the Council. The interim report was published on 10 December 2010 7 and the Commission submitted its report communicating these results in April 2011 8 .

The Eurostars Decision also required a final evaluation at the end of the programme, and for the results again to be communicated to the European Parliament and the Council.

The Makarow group, a group of independent experts chaired by Marja Makarow, vice-president of the Academy of Finland, carried out the final evaluation, which was published in November 2014. 9 The group used qualitative and quantitative methods to assess the relevance, efficiency, impact and sustainability of the programme. Compared with the interim evaluation, the numbers of applications, applicants and funded projects are about three times greater. In contrast to the interim evaluation, the Makarow group was therefore able to make quantitative calculations and carry out extensive econometric analyses.

The Makarow group’s main conclusion is that: ‘The Eurostars Joint Programme has succeeded in accelerating the growth and innovative outputs of R&D-performing SMEs. However, several aspects of governance and managerial implementation need to be improved.’

The main purpose of the current report is to communicate the results of the final evaluation to the European Parliament and the Council, as required by the Eurostars Decision. This report provides a short summary of the Commission’s interim evaluation and the 2011 report (section 2) followed by the Makarow group’s main findings and recommendations (section 3). In line with the 2011 report, the Commission also submits its observations on the main aspects of the final evaluation (section 4). Based on the Makarow group’s report, these observations are focused on those aspects of Eurostars that the Commission considers can be improved, in particular those that were already raised in the interim evaluation.

The Commission also takes account of the Eurostars-2 Decision and the fact that several recommendations from both the interim and final evaluation have already been taken on board in this Decision and in the new delegation agreement.

2. Interim evaluation and the Commission's 2011 report

Article 13(2) of the Eurostars Decision provides that: ‘An interim evaluation of the Eurostars Joint Programme shall be carried out by the Commission two years after the start of the Eurostars Joint Programme and shall cover progress towards the objectives set out in Annex I. The evaluation shall also include recommendations on the most appropriate ways to further enhance scientific, management and financial integration and assess the ability, of R&D-performing SMEs in particular, to access the Eurostars Joint Programme and the quality and efficiency of its implementation. The Commission shall communicate the conclusions thereof, accompanied by its observations and, where appropriate, proposals for the amendment of this Decision, to the European Parliament and the Council.’

The 2009 delegation agreement between the European Commission and the EUREKA Secretariat set out guidance on applying the criteria of the Eurostars Decision and specified explicit questions to be addressed in the evaluation. 10  

A group of independent experts 11 appointed by the Commission submitted the interim evaluation report in December 2010. The group concluded that: ‘Eurostars is a good programme, which meets its objectives and adds value to European R&D-performing SMEs. For this reason, the group of independent experts believes that Eurostars should not only be sustained but, preferably, its budget should be increased in the future. However, in spite of good progress, some scope for further improvement also remains.’

The interim evaluation took place in 2010 and the report contains data from the first four cut-off dates: there were two cut-offs in 2008, one in 2009 and one in 2010 12 . A total of 1127 applications from 3790 applicants had been received for the first four cut-offs. For the funding stage, the report only contains data from the first three cut-offs. Funding was approved for 264 projects with total projects' cost of EUR 386 million. Only 14 projects had been completed. In the two years since its inception, up to spring 2010, Eurostars mobilised a total earmarked budget of public funding of EUR 231.1 million, of which EUR 173.5 million came from the participating countries and EUR 57.6 million from the EU.

For the first two cut-offs, the average time-to-contract 13 was 11.4 months. This time varied significantly, from a minimum of 5.3 months to a maximum of 26.8 months in the first cut-off period (average 11.8 months), and from a minimum of 6.3 months to a maximum of 17.2 months in the second (average 11.0 months). The annex to the delegation agreement (point 5, ‘Expected results and indicators’) gives a target for time-to-contract of six months, to be achieved by the mid-term of the programme.

The group considered the three main strengths of the programme to be as follows:

1) with a large number of applications, demand for the programme has exceeded original expectations and it has mobilised predominantly R&D-performing SMEs with close-to-market, bottom-up projects;

2) programme management has improved over time in organising the necessary logistics and infrastructure and the degree of satisfaction of national authorities and participants is increasing; and

3) central evaluation is considered as best practice in terms of clear and transparent organisation and timely accomplishment.

However, the group pointed to five areas where there is some scope for improvement:

1) harmonising and synchronising national procedures;

2) attracting more applicants, in particular SMEs with no prior experience of international collaboration;

3) improving the geographical balance of technical experts, and the balance between technical and market expertise within the pool of experts;

4) increasing visibility in tandem with ensuring the ability to fund more projects; and

5) continuing efforts towards the organisation of a virtual common pot.

In its interim report to the European Parliament and the Council, the Commission stated that the group’s evaluation report covered all the aspects mentioned in Article 13(2) of the Eurostars Decision, and provided additional facts, comments and recommendations. The Commission considered the whole report to be an integral part of the interim evaluation process set out under Article 13(2). In its comments, the Commission highlighted only the most relevant findings and recommendations on the way forward.

The Commission’s observations on the interim evaluation — which are still relevant in light of the final evaluation — concerned in particular how to further enhance scientific, management and financial integration, and the levels of quality and efficiency in implementing Eurostars.

The Commission agreed with the group’s recommendations with a view to further accelerating the integration process. The Commission invited Eurostars countries to take the programme’s integration objective into account through greater harmonisation and simplification of funding eligibility rules and alignment of financing and disbursement practices. The Commission stressed that the Eurostars eligibility criteria should be universally applied across all participating countries; no country should be allowed to impose eligibility conditions going beyond those of Eurostars. The Commission also agreed with the proposal to test a partial (10 % of total earmarked funds) ‘real’ common pot approach.

Regarding the quality and efficiency of implementation, the Commission supported proposals to improve the central evaluation process to ensure impartiality, quality and timeliness. Experts must have the necessary expertise, and better feedback from evaluators to applicants would ensure transparency. The Commission also agreed with recommendations to reduce the time-to-contract.

The Council gave its views on the interim evaluation in its conclusions of 31 May 2011 14 . The Council ‘WELCOMES the view of the Group that Eurostars is well aligned with the objectives of Europe 2020, complements well the opportunities offered to SMEs in FP7 for international cooperation and has proven to be attractive to the target group by reaching successfully European R&D-performing SMEs’; ‘WELCOMES the recommendation of the Group to continue Eurostars beyond 2013; and WILL CONSIDER its continuation in the overall context of the future Common Strategic Framework for Research and Innovation funding’ 15 .

3. Final evaluation

3.1.Introduction

Article 13(3) of the Eurostars Decision states that: ‘At the end of the Eurostars Joint Programme, the Commission shall conduct a final evaluation of the Programme. The results of the final evaluation shall be presented to the European Parliament and the Council.’

The Commission drafted terms of reference setting out the mandate, scope and objectives of the final evaluation. Together with the Eurostars Decision and the delegation agreement, the terms of reference provided the basis for the Makarow group regarding approaches to be taken and questions to be addressed.

The data presented by the Makarow group covers all 10 cut-off dates from 2008 to 2013. The EUREKA Secretariat received 3548 applications comprising 11733 applicants. The overwhelming majority of applicants were SMEs (72 %) and the average consortium size was 3.3 participants. Funding was approved for 783 applications for a total of EUR 1 130 million. The number of applications continued to increase throughout the duration of the Eurostars programme. However, the number of applications approved for funding did not rise accordingly, meaning that the success rate 16 declined from 42 % in 2008 to 17 % in 2013. Total estimated public funding for 2008-2013 was EUR 472 million, giving a proportion of EU funding (EUR 100 million) to national funding (EUR 372 million) of 26.9 % (or 21.2 % of the total public funding.

The average time-to-contract has improved compared with the situation at the time of the interim evaluation. For example, in the third cut-off period (2009), the average time-to-contract was 435 days, whereas by the eighth cut-off in 2012, it had fallen to 282 days. However, the time-to-contract is still fairly long in a significant number of countries. The Makarow group considers that the programme’s time-to-contract must be improved urgently, as the existing considerable differences between participating countries calls the whole system into question.

3.2.Methodologies

The Makarow group used qualitative and quantitative tools for the final evaluation. The main methods were: desk research; interviews with stakeholders and SMEs; an online survey of SMEs that had applied for Eurostars; and quantitative analyses.

Through desk research, the group gathered documents from various stakeholder groups and analysed the data and information relevant for the Eurostars evaluation criteria. This desk research enabled the group to identify major issues, challenges and questions on the implementation of Eurostars. The source material included legislative and cross-cutting documents, minutes of the EUREKA high-level group meetings, statistical data and data hosted by the EUREKA Secretariat.

The Makarow group held interviews in Brussels with two representatives of the Commission, three representatives of EUREKA and with the EUREKA Secretariat’s top management. Individual group members conducted face-to-face or telephone interviews with 20 representatives of national governing and administrative bodies of Eurostars and with the chief executive officers of 26 SMEs, operating in 12 different countries. The group drew up guidelines to structure the interviews, the guidelines being adjusted according to the function of the interviewees. All interviews were documented in writing. The interviews covered a wide range of topics.

As part of the evaluation, an online survey asked participants about their experience with Eurostars. The target group of the online survey was restricted to R&D-performing SMEs and other SMEs. The online survey ran from 9 May to 10 July 2014. The survey reached out to 6 620 SMEs through emails linking to an online questionnaire. Two different questionnaires were prepared: the first targeted funded projects, the second targeted non-funded projects. A total response rate of 46 % was obtained (72 % for funded and 39 % for the non-funded SMEs).

To understand the extent of the Eurostars impact, the Makarow group developed an econometric counterfactual impact evaluation. The econometric study focused primarily on job creation in participating SMEs and on innovation output as measured by patent applications. With respect to data gathering, this required that the EUREKA Secretariat’s SME applicant data be matched to the Bureau van Dijk’s Amadeus database, from which information on employment figures were gathered. In addition, the EUREKA Secretariat's SME applicant data were matched with the European Patent Office Worldwide Patent Statistical Database. The resulting database brings together applicant information and time series data on employment and patent applications, enabling estimations of econometric treatment effects to be made.

3.3.Findings and assessment

• Target group and scope of the programme

The Makarow group points out that, according to participating SMEs and the governing and administrative bodies of Eurostars, the programme is relevant to the target group’s needs and objectives. The high level of demand from SMEs is considered an indication of this ‘strategic fit’. Almost all participating SMEs plan to apply to Eurostars again, as the programme corresponds to their needs and objectives. R&D-performing SMEs count for the overwhelming majority of participants in Eurostars, and received about 75 % of the funding. In addition, the programme was successful in addressing young, small and micro firms.

Participating SMEs already have experience in national and international funding programmes, and the majority already have pre-existing international links. However, the Makarow group found that applications involving university or public research organisations have a higher chance of being funded. This may be because the presence of a university or public research organisation signals the high quality of the proposal, or that those organisations are more experienced in preparing application documents, or that those applications provided the best balance between scientific excellence and market knowledge.

Based on its bottom-up nature, Eurostars is, according to the group, able to address a wide-variety of innovative technologies, ranging from information and communication technology and its applications, and medical biotechnology, to (renewable) energy and steering and control technologies.

• Governance

The Makarow group notes that the EUREKA Secretariat is generally acknowledged as a successful dedicated implementation structure. However, there is a need to improve managerial and administrative practices, enhance the qualifications of staff, develop and implement a marketing and promotion strategy, and develop and maintain a well-structured information management system and database for use by the Secretariat and the other Eurostars governing and administrative bodies.

The group considers that the Eurostars governance system functions properly and that interaction between national and central administrative bodies is effective. However, the international dimension of Eurostars required a complex governance system to be set up.

The EUREKA high-level group regularly discusses various issues critical to the success of Eurostars without necessarily reaching a consensus for action or agreeing binding rules for all countries. Synchronisation and harmonisation of funding, eligibility rules and the adequacy of national budgets have been intensely analysed and discussed. The Makarow group considers that there is a need to increase the high-level group’s ownership of and involvement in Eurostars. This is critical to facilitate synchronisation and harmonisation. Similarly, the national funding bodies’ involvement remains limited in the governance structure; they should be more involved in order to achieve greater synchronisation and harmonisation.

Finally, Eurostars has strengthened the SME presence within EUREKA and contributed to a much higher visibility of their contribution to European competitiveness and innovation. In this way, Eurostars has had a significant and long-lasting positive effect on the EUREKA network.

• Management and operations

The Makarow group considers that the virtual common pot has worked well and is the preferred funding mechanism, since it attracts countries to contribute to Eurostars. In response to the programme’s success, several countries increased their contribution rates significantly beyond the initially agreed level. However, the full benefits of the virtual common pot have not yet been exploited in Eurostars, since some countries, in particular Germany and Spain, do not allocate a sufficient level of funding for the programme. Because of this, some good projects selected fall from the ranking list, which in turn leads to frustration among applicant SMEs.

The group reports that synchronisation of funding is still the most important challenge facing Eurostars, although some improvements have been achieved. In some cases, participating SMEs face great difficulties because of the longer time-to-contract. There are also parallel application and evaluation requirements and dual reporting obligations applied by some national funding bodies.

The other major challenge facing Eurostars is the harmonisation of funding rules. The high-level group considers that this challenge should be addressed through a top-down process at the initiative of national ministries and the EU. National project coordinators think that full harmonisation is not possible: for example, they cannot change the funding percentage or the financing upper limit for each SME. Another issue is the need for greater harmonisation of subcontracting rules for universities and public research organisations.

The Eurostars central evaluation process has worked well. However, the Makarow group considers it necessary to improve the transparency and feedback mechanism and to streamline the process in order to decrease the time from the initial application to the availability of an evaluation report. The time-to-contract was rather long at the start of the Eurostars programme, but decreased considerably during the first phase.

Although the number of technical experts and of those with market experience has increased considerably since 2012, there is still room to improve geographical and gender balance. National project coordinators would also like to be involved in the process of improving the experts’ database. SMEs are satisfied with the level of information they receive about the evaluation of their projects. They deem the reporting workload and the monitoring process to be appropriate, and they also find the evaluation process clear, transparent and timely.

• Funding

The Makarow group highlights that one indicator of the success of Eurostars was the significant increase in the number of applications and hence the increasing demand for public funding. The annual number of applications submitted increased from 215 in 2008 to 948 in 2013. During the lifetime of Eurostars, however, this resulted in a decline in the proportion of submitted applications that were funded. The success rate fell from 42 % in relation to the first cut-off date to 17 % for the tenth. The proportion of projects meeting the quality threshold that were approved for funding also declined, from 68 % for cut-off 1 to 55 % for cut-off 10; in short, an increasing proportion of high-quality projects were not funded. The Makarow group considers that the increase in the number of applications per year is ‘remarkable’ and ‘impressive’.

National budgets for Eurostars still vary enormously. This reflects the relative size of the country and the organisation of the national public funding system, and also demand from national SMEs for Eurostars funding. These differences had a significant negative impact on the functioning of the virtual common pot system. The size of funding per participant is judged as adequate by most participants and programme administrators.

The Makarow group considers that Europe’s financial and economic crisis had a negative effect on national funding for Eurostars in the majority of countries. The number of projects above threshold decreased considerably owing to lack of funds, and for this reason some countries did not participate in the latest calls for tender. The countries that were not short of funds also reported that they were affected negatively by the crisis, since the SMEs that they supported were partnered with counterparts in countries that were having funding difficulties.

• Uptake of recommendations from the interim evaluation report

The Makarow group assessed the uptake of recommendations from the interim evaluation report. Of 21 policy recommendations to be implemented by 2013, the Makarow group considers that seven recommendations have been fully implemented, ten have begun to be — or been partially — implemented, while four have not been implemented at all.

These four recommendations concern:

1) setting a global deadline for the signature of grant agreements;

2) giving the EUREKA high-level group an active role in monitoring the time-to-contract and in ensuring that countries that lag behind streamline their processes;

3) experimenting with the approach of allocating 10 % of total earmarked funds to a real common pot; and

4) having the EUREKA Secretariat initiating an activity to explore the integration of EUREKA cluster initiatives in view of learning and implementing integration lessons that could be applied to promote the integration of Eurostars with national R&D support initiatives.

3.4.Recommendations

The Makarow group’s report contains 28 recommendations.

• Target group and scope of the programme

1) The EUREKA Secretariat should develop an action plan to promote Eurostars participants’ contacts with R&D and innovation intermediaries mostly involved in the project preparation phase.

2) The Secretariat should promote linkages between Eurostars participants and different types of EUREKA projects in order to facilitate Eurostars projects’ connections with clusters, umbrellas and other thematic or individual projects.

3) The Secretariat should organise brokerage events, such as pitch competitions, between Eurostars participants and financing institutions to increase the possibilities of successful projects to receive follow-up funding from the private sector. A dissemination strategy should be developed to target financiers, other SMEs, and national and international government bodies.

4) The Secretariat should raise awareness among the Eurostars beneficiary SMEs of the importance of protecting the intellectual property rights of their innovations.

5) The Secretariat and national project coordinators should ensure that the programme also reaches SMEs that do not have pre-existing international links, by designing and implementing a marketing and promotion strategy.

6) The Secretariat and the high-level group should agree on the definition and development time of ‘near market R&D’ projects for Eurostars, which is neutral vis-à-vis the specificities of particular technologies and markets.

7) The European Commission and the Secretariat should check the possibilities for linking Eurostars and the new Horizon 2020 SME instrument to capitalise on the potential growth and competitive results of Eurostars SME participants.

• Governance

1) Decisions taken by the high-level group should be translated without delay into implementation plans with concrete objectives, tasks, and allocation of responsibilities, using a SMART (specific, measurable, attainable, relevant and time-based) approach.

2) The high-level group should identify countries that impose parallel application and evaluation requirements as well as dual reporting obligations to Eurostars projects, and agree with them to eliminate the national processes. The high-level group should ensure that this is implemented before the spring cut-off of Eurostars-2 in 2016.

3) The Secretariat should improve the transparency of the central evaluation process and feedback mechanisms to applicants, national funding bodies and national project coordinators. The coordinators should have access to the consensus reports of the independent evaluation panel, as well as to the individual reports of the technical evaluators, to be able to give detailed feedback to applicants.

4) The Secretariat should ensure a more balanced geographical distribution of technical experts in close cooperation with national project coordinators. Attention should be paid to achieve gender balance as well. In addition, the Secretariat should expand the pool of technical experts with appropriate knowledge of markets.

• Management and operations

1) The EUREKA Secretariat should maintain the frequency of two cut-offs per year.

2) The Secretariat should decrease the time from application (cut-off date) to evaluation to 4-5 months so that results of the central evaluation will be available to national project coordinators and applicants within five months (150 days) at the latest. Every national funding body needs to sign the funding contracts within 2-3 months. It is recommended that the EUREKA Secretariat, together with the chair of the high-level group and the heads of national funding bodies, reviews the programme management processes of those national funding bodies that risk not meeting the deadline, and agrees with them the steps to be taken to shorten the process.

3) A standard consortium agreement should be provided to applicants in order to speed up the contractual demands of project participants.

4) The Secretariat should establish and maintain a robust central Eurostars database to collect, store and process reliable data on projects and beneficiaries. It should introduce quality checks of data for the application forms as well as for project progress, final and market impact report forms. The deadline for submission of the final reports should be reduced to one year and market impact reports should be available within two years of the end of the project.

5) The Secretariat should modify the application form and the project progress, final and market impact report forms. Detailed revisions of the current specific questions and headings in the forms, as well as re-ordering, deletion and addition of a number of questions, are required to improve the logic of the forms. Correspondingly, the guidelines for these forms should be adjusted accordingly.

6) The Secretariat should develop a dedicated application form and corresponding guidelines for resubmissions. The resubmissions need to include an explanation on what has been changed, especially in response to comments from previous evaluations received via the national project coordinators.

7) The Secretariat should design and implement an institutional development plan to strengthen its programme and organisational management practices in order to be able to improve the implementation of Eurostars in line with international best practices, and to hire, retain and motivate qualified staff experienced in the management of R&D and innovation programmes. It is recommended that the Secretariat receives mentoring from the network’s national funding bodies with relevant competencies.

8) The Secretariat should introduce a unique identifier for all organisations, not only for SMEs and large firms but also for universities, public research organisations, and other partners of R&D-performing SMEs. If available, each organisation should provide their Participant Identification Code under Horizon 2020. The application form should include information about whether the project was submitted previously to another national or EU programme.

9) The Secretariat should collect the information from all countries about the time-to-moneyand submit this information to the high-level group, and agree on a binding deadline for collection and submission.

10) Information on the exact start and end dates of the projects needs to be included in the central Eurostars database. National project coordinators should without delay inform the EUREKA Secretariat on signed grant agreements, including start and end dates of the projects.

11) The Secretariat, the high-level group and national funding bodies should ensure that the time-to-contract will not exceed 7 months (210 days) as agreed for Eurostars-2 in the Budapest document.

12) From the spring cut-off of 2016 onwards, the Secretariat, the high-level group and national funding bodies should agree that, for at least half of the projects, the grant agreements between national funding body and programme participants will be signed within 210 days of the cut-off date. For 90 % of the projects, all grant agreements should be signed within one year from the cut-off date.

13) The Secretariat and national project coordinators should jointly define and develop tools to facilitate information exchange between them, improve the quality and quantity of information available from and for both sides, allow monitoring of all project phases, and reduce the administrative cost of bilateral information exchanges.

14) The European Commission should consider not granting funding to those Eurostars projects for which all grant agreements have not been signed within one year of the cut-off deadline unless the delay in signing a funding contract is caused by members of the consortium funded.

• Funding

1) The high-level group is strongly encouraged to agree on a minimum baseline for funding rules to be harmonised and to ensure that each participating country conforms to this. The maximum funding rates should be the same for each type of partner in all countries. The process should be closely followed up by the high-level group and be completed by the first cut-off in 2016.

2) The funding of universities and public research organisations should be harmonised so that they participate as partners of the foreign R&D-performing SMEs and not only as subcontractors of R&D-performing SMEs from the same country.

4. Commission’s observations

4.1.Introduction

The Commission considers that the Makarow group report has addressed the main aspects that were to be covered by the evaluation, as described in the Eurostars Decision, the delegation agreement and in the terms of reference.

As Member States and other participating countries have increased their funding contribution to Eurostars — estimated at EUR 372 million — the EU contribution of EUR 100 million, at 26.9 %, fulfils the requirement of the Eurostars Decision that EU funding should not exceed one third of the contributions of Member States and other participating countries.

The Eurostars Decision refers to the importance of SMEs for European growth and competitiveness. The high and increasing number of applications for funding from Eurostars, and the results obtained by the supported SMEs, indicates that the programme is relevant for the growth of R&D-performing SMEs.

Since the start of the programme in 2008, several aspects of Eurostars improved up to the end of 2013. For example, time-to-contract was shortened and harmonisation and synchronisation improved. The Commission welcomes this and appreciates the efforts made by the EUREKA Secretariat and national funding bodies.

The Makarow group’s main conclusion is that Eurostars has succeeded in accelerating the growth and innovative outputs of R&D-performing SMEs. However, the group also considers that aspects of governance and managerial implementation need to be improved.

The first part of the main conclusion is substantiated by extensive econometric impact analysis. This shows that the employment growth rate of R&D-performing SMEs funded by Eurostars was nearly twice as high as that of applicant SMEs that were not funded. Concerning the second part, the group concluded that the lack of harmonised funding rules and synchronised national processes led to inefficiencies. The group also considers that the effectiveness of central governance, administration and operations needs to be improved.

The Commission finds that the findings, assessment and recommendations of the final evaluation have been convincingly substantiated. The Commission agrees with the main conclusion of the Makarow group.

Several of the recommendations from the interim evaluation and the final evaluation have already been taken on board in the Eurostars-2 Decision (2014-2020). 17

As the Makarow group’s report constitutes an integral part of the evaluation process, the Commission does not comment in the following observations on every detail of the report. As the Commission’s focus is forward-looking, these observations are mainly concerned with aspects of Eurostars for which the Commission considers there is the greatest scope for improvement, i.e. the second part of the Makarow group's main conclusion.  

4.2.Target group and scope of the programme

Both the interim evaluation and the Makarow group report indicate that the EUREKA Secretariat and national project coordinators should ensure that Eurostars also reaches SMEs that do not have pre-existing international links. The Commission agreed with this recommendation in 2011, and the Commission still agrees in 2015. Although this recommendation is addressed to both the Secretariat and the coordinators, several other recommendations concerning the promotion of — or raising awareness about — Eurostars are addressed only to the Secretariat (cf. recommendations 3, 4, 5 and 7 above). The Commission agrees with these recommendations. However, although the EUREKA Secretariat has a pivotal role, participating Eurostars countries and national funding bodies also have a responsibility for reaching more and better R&D-performing SMEs with a potential for growth.

The Makarow group recommends that the Commission and the Secretariat should check the possibilities for linking Eurostars and the new Horizon 2020 SME instrument (cf. recommendation 6). Although it is unclear what ‘linking’ means, the Commission analysed the strategic positioning of the different instruments when the proposal for Eurostars-2 was presented. 18 The Commission considers that the support they provide occurs at different phases in the development of the SMEs research and business ideas with relation to the market. Eurostars is different and complementary — not supplementary — to the SME instrument in Horizon 2020. The delegation agreement also states that national funding bodies should take effective measures to avoid any double funding of final beneficiaries’ projects through other EU funding sources. That Eurostars and the SME instrument have different target groups needs to be clearly communicated by the EUREKA Secretariat, the European Commission and the Executive Agency for Small and Medium-sized Enterprises.

4.3.Governance

The Makarow group notes that the EUREKA high-level group regularly discusses various issues critical to the success of Eurostars without necessarily reaching a consensus for action or agreeing binding rules for all countries. The group considers it necessary to increase the high-level group’s ownership of and involvement in Eurostars.

The Commission agrees with this assessment and the recommendation that decisions by the high-level group should be taken under a SMART (specific, measurable, attainable, relevant and time-based) approach (recommendation 8). One concrete example of such an approach is recommendation 9 for the elimination of parallel application and evaluation requirements and dual reporting before the first cut-off date in 2016. The Commission fully supports this recommendation.

The Makarow group also strongly encourages the EUREKA high-level group to agree on a minimum baseline upon which to harmonise funding rules (recommendation 27). The group considers that this should be completed by the first cut-off date in 2016. The Commission considers that improvements still remain to be made regarding harmonisation of national funding rates and funding rules. The Commission therefore fully agrees with this recommendation.

4.4.Management and operations

Time-to-contract — and the differences in time-to-contract across participating countries — was considered in the interim evaluation and has also been analysed by the Makarow group.

Time-to-contract is determined at two levels. First, the central evaluation is carried out and coordinated by the EUREKA Secretariat. Second, the funding allocation and contract negotiations take place at national level. Both the interim evaluation and the Makarow group found that the quality of the central evaluation is quite good and runs smoothly.

However, the Makarow group recommends several concrete measures at both levels to reduce time-to-contract (cf. recommendations 14, 15 and 16 above). The Commission agrees with these recommendations, but it seems clear that the main problem is the huge differences at national level. The Commission considers that the EUREKA high-level group and Secretariat, and the national funding bodies, should give priority to reducing these differences.

The Makarow group also recommends that the Commission should consider withholding grant funding for any Eurostars project for which all grant agreements have not been signed within one year of the cut-off deadline (recommendation 17 above).

The Budapest document stated that, for Eurostars-2, national funding bodies ‘shall be aiming at a time-to-contract of 7 months after the cut-off deadline, but preferably faster’. The 2014 Eurostars-2 Decision set an objective to increase efficiency of public funding ‘by aligning, harmonising and synchronising the national funding mechanisms of participating states’ (Article 3). The Decision also clearly states that the Union’s financial contribution is conditional upon demonstration by participating states that they have set up Eurostars-2 in accordance with the objectives laid down in Article 3. The same provisions are also to be included in bilateral agreements signed between the EUREKA Secretariat and national funding bodies. Such agreements ‘shall include the rules governing the transfer of the Union’s contribution and the minimum operational targets and national progressive milestones for further integration and synchronisation of national programmes, including a shorter time-to-grant in accordance with Regulation (EU, Euratom) No 996/2012 and Regulation (EU) No 1290/2013’ (Annex I, paragraph 11). Horizon 2020 also contains clear rules regarding time-to-grant: applicants are to be informed within five months of the outcome of the scientific evaluation of their applications, and grant agreements are to be signed or grant decisions notified within three months from the date of informing applicants they have been successful. 19 The Commission therefore fully supports the Makarow group’s recommendation 17 and calls on participating states to respect their own commitment to reduce the time-to-contract.

The Makarow group also made several recommendations regarding the EUREKA Secretariat database, including how data is collected and entered into the database (cf. recommendations 19 — 24 above).

The Commission agrees that the EUREKA Secretariat application process and database should be improved. The forms to be filled in by applicants must be as logical and easy to understand as possible. Logic checks for fields requiring arithmetic completion should be in place to reduce the probability of entering wrong values. A uniform approach regarding terminology should be introduced, both in the forms and in the guidance to users. The technological area classification should also follow the nomenclature of the EU’s statistical classification of economic activities (NACE Rev.2) in order to provide a common basis for the comparative evaluation of different EU funding schemes.

4.5.Role of universities and public research organisations

The Makarow group recommends that the funding of universities and public research organisations be harmonised so that they participate as partners and not only as subcontractors (cf. recommendation 27).

The Commission understands this recommendation to mean that a project or consortium could, for example, consist of an SME in one country, a university in a second country and a public research organisation in a third country. The Commission considers that this would probably lead to universities and public research organisations playing a bigger role in Eurostars projects and that a bigger part of the budget would go to such institutions.

The Eurostars Decision provides that the main activity in Eurostars consists of R&D-activities led by one or more R&D-performing SMEs. Although universities, public research organisations and large companies may also participate in the programme, the target group for Eurostars is clearly R&D-performing SMEs.

However, Horizon 2020 dedicates substantial funding to intra-European research collaborations between actors belonging to the Knowledge Triangle of research, business, and higher education, through the Marie Skłodowska-Curie actions and the European Institute of Innovation & Technology.

The Commission therefore does not support this recommendation.

5. Conclusion

The Eurostars programme has shown good results in terms of accelerating the growth and innovative outputs of R&D-performing SMEs. The final evaluation provides evidence that there is a need for Eurostars and that the programme is reaching its target group.

However, the Commission agrees with the Makarow group that the most important challenge for Eurostars is to synchronise funding and to harmonise funding rules. Although some progress has been made, it is essential therefore, for the long-term effectiveness of the programme, that the EUREKA high-level group and Secretariat, and the national funding bodies, make increased efforts to accelerate synchronisation and harmonisation under the Eurostars programme.

(1) Decision No 743/2008/EC of the European Parliament and of the Council of 9 July 2008 on the Community’s participation in a research and development programme undertaken by several Member States aimed at supporting research and development performing small and medium-sized enterprises. (OJ L 201, 30.07.2008, p.58-67): http://eurlex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2008:201:0058:0067:EN:PDF .
(2) All 28 Member States plus five other countries (Iceland, Israel, Norway, Switzerland, and Turkey) have participated in Eurostars (2008-2013). As of February 2008, 20 Member States had started their participation. Belgium, Croatia, Italy, Latvia, Luxembourg and United Kingdom have participated since November 2008, Bulgaria since February 2010, and Malta since March 2011.
(3)

     Decision No 553/2014/EU of the European Parliament and of the Council of 15 May 2014 on the participation of the Union in a research and development programme jointly undertaken by several Member States aimed at supporting research and development-performing small and medium-sized enterprises (OJ L 169, 07.06.2014, p. 1-13):

http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014D0553&from=EN .

(4) Under Eurostars 2008-2013, R&D-performing SMEs are defined as SMEs dedicating at least 10 % of their staff (full-time equivalents), or 10 % of their turnover, to R&D activities.
(5) The definition of SMEs is set out in Recommendation 2003/361/EC of 6 May 2003 concerning the definition of micro, small and medium-sized enterprises (OJ L 124, 20.05.2003, p.36): http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2003:124:0036:0041:EN:PDF . The main factors determining whether a company is an SME are the number of employees (<250) and either turnover (<€50 m) or balance sheet total (<€43 m).
(6) EUREKA is an intergovernmental network launched in 1985 to support market-oriented R&D and innovation projects by industry, research centres and universities across all technological sectors. It is composed of 41 members, including the European Union represented by the Commission: http://www.eurekanetwork.org/ .
(7) http://ec.europa.eu/research/evaluations/pdf/archive/fp7-evidence-base/other_fp7_panel_evaluations/eurostars_programme_interim_evaluation.pdf#view=fit&pagemode=none .
(8) COM(2011) 186 final Brussels, 8.4.2011: http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52011DC0186&from=EN .
(9) https://ec.europa.eu/research/sme-techweb/pdf/ejp_final_report_2014.pdf .
(10) Eurostars Joint Programme delegation agreement 30-CE-0270684/00-14 (signed in June 2009), Annex 1, point 4, pages 11 and 12. The Eurostars Decision (Article 4) states that the arrangements for the EU financial contribution are to be established by means of a general agreement between the Commission and the EUREKA Secretariat.
(11) The group was chaired by Ms Anne Laperrouze (former MEP). The other members of the group were: Professor Erkko Autio (Rapporteur), Professor Maja Bucar, Georg Licht, Professor Jose Molero and Professor Lena Tsipouri.
(12) Eurostars is open to funding applications on a continuous basis, but the Secretariat sets cut-off deadlines for submitting applications. The central evaluation process starts after each cut-off.
(13) Time-to-contract is defined as the time between the cut-off date and the date of signature of the grant agreement.
(14) The Council conclusions on the interim evaluation of Eurostars (Brussels, 1 June 2011, RECH 144, COMPET 226, MI 291): http://register.consilium.europa.eu/doc/srv?l=EN&f=ST%2011030 %202011 %20INIT .
(15) http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ:L:2013:347:FULL&from=EN In the Regulation establishing Horizon 2020, the European Parliament and Council decided to support research-intensive SMEs through the Eurostars programme. See Annex I, point 3(3)(b), p. 147, of Regulation (EU) No 1291/2013 of the European Parliament and of the Council of 11 December 2013 establishing Horizon 2020 — the Framework Programme for Research and Innovation (2014-2020) and repealing Decision No 1982/2006/EC (OJ L 347, 20.12.2013, p. 104): .
(16) The success rate is defined as the total number of applications approved for funding divided by the total number of applications.
(17) For example, the Eurostars-2 Decision states that: ‘As part of the improvements from the previous Eurostars programme, Eurostars-2 should head towards shorter time-to-grant, stronger integration and lean, transparent and more efficient administration to the ultimate benefit of research and development-performing SMEs.’
(18) Brussels, 10.7.2013, SWD(2013) 242 final.
(19) http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ:L:2013:347:FULL&from=EN See Article 20 of Regulation (EU) No 1290/2013 of the European Parliament and of the Council of 11 December 2013 laying down the rules for participation and dissemination in ‘Horizon 2020 — the Framework Programme for Research and Innovation (2014-2020)’ and repealing Regulation (EC) No 1906/2006 (OJ L 347, 20.12.2013, p. 81): .