REF seeks applications for the position of an External Evaluator


Impact Evaluation of Roma Education Fund

Currently, the Roma Education Fund is seeking an external evaluation to provide deeper understanding of the impact, gaps and challenges in funding Roma education. The results of the evaluation will be used to inform donors, board members, and to influence the future strategy of the organization.


Impact Evaluation of Roma Education Fund

Currently, the Roma Education Fund is seeking an external evaluation to provide deeper understanding of the impact, gaps and challenges in funding Roma education. The results of the evaluation will be used to inform donors, board members, and to influence the future strategy of the organization.

 
The deadline for submitting applications is 22 May, 2012.
  

– Terms of Reference –

1.    Background Information

The Roma Education Fund (REF) was founded by the Open Society Foundations (OSF) and the World Bank in the framework of the Decade of Roma Inclusion in 2005.  Its mission and ultimate goal is to close the gap in educational outcomes between Roma and non-Roma. In order to achieve this goal, the organization supports policies and programs which ensure quality education for Roma, including the desegregation of education systems.
Presently, donors include OSF, the World Bank, the Austrian, British, Finnish, German, Spanish, Swedish, and Swiss governments; private companies such as Putumayo World Music; large foundations such as the Network of European Foundations, the World Bank Community Fund, Tempus Foundation Hungary, and the Foundation Remembrance, Responsibility and Future (EVZ); as well as several individuals. In addition, the European Commission and the Romanian Structural Funds-Management Authority have recently become REF donors.

REF provides additional assistance to countries to improve the educational outcomes of Roma. REF is designed to spur changes in government policies and programs, build upon the lessons learned from successful interventions and pilot programs, and disseminate knowledge on issues affecting Roma in education. Active participation of the Roma community in all REF-funded activities is a prerequisite.

REF is not designed to substitute for government resources but rather to supplement and push ahead efforts by governments, international organizations and civil society.. REF funds projects and programs designed by public entities – including local, regional, and national governments —  private or non-profit entities, and combinations of public and private organizations.  
REF’s work to date has been divided into five major Programs, all of which are coordinated at the senior level from Budapest.

These five Programs are as follows:

(1)    Project Support Program
(2)    Scholarship Program
(3)    Policy Development and Capacity Building Program
(4)    Communication and Cross Country Learning Program
(5)    Reimbursable Grant Program

The evaluation should assess the impact (intended and unintended) of REF’s cumulative interventions (via the five main Programs listed above) on REF’s mission:
•     To reduce the educational gap between Romani and non-Romani children by providing better access to quality education.
•    To promote Roma inclusion in all aspects of the national education systems of countries participating in the Decade of Roma inclusion, as well as others wishing to join this effort.
•    To reduce the segregation of Romani children in education systems.

What changes in law, policy, practice and outcomes related to Roma education can be directly and/or indirectly attributed to REF interventions?  

2.    Purpose, Objectives, and Scope of the Evaluation

As part of REF’s quality improvement and accountability to our donors, we have conducted and commissioned a number of internal and external evaluations of our work. These evaluations have attempted to assess the impact of REF’s efforts, to demonstrate the impact of investments for our current donors, and to demonstrate to potential donors the value of investing in REF. The first external evaluation of REF at the organizational level was prepared in October 2008 and covered the period up to 30 June of the same year.
Because four years have passed since this evaluation, REF and some of its main donors are commissioning a new impact assessment evaluation. Beyond the need for a current evaluation of work that has been done since 2008, REF has become increasingly involved in high level policy work. Thus, there is a strong interest on the part of both REF and its donors in evaluating the efficacy of this policy work and its impact on our mission, objectives and activities.

The evaluation should identify where REF has contributed to the reform of government law, policy and practice that can be expected to contribute to improved educational outcomes and educational inclusion for Roma; whether improvements in educational outcomes can be directly attributed to these reforms; and which instruments and approaches of REF were most effective for reaching its mission objectives. Thus, the evaluation may include, for example, the identification of policy change objectives in REF’s work in selected countries and whether these objectives were achieved, a description of which of the educational interventions supported or implemented by REF have worked and which have been less successful. If possible, evaluators might also examine the cost efficiency of REF’s direct policy interventions and policy interventions through its grant Program activities.
 

Following are the main questions of interest to REF and its donors; we would like to see an evaluation proposal that addresses many if not all of them:

•    How and to what degree has REF’s work  resulted in changes in law, policy or practice designed to increase access to education and improved educational outcomes for Roma across Europe?
•    To what degree has REF’s work contributed to measurable, increased access to education or improved educational outcomes for Roma in fact?  
•    How effective and relevant  has REF’s policy advocacy work been in furthering its mission objectives?  How significant is the policy advocacy work in financial terms?  How has it been incorporated into other work supported by REF (such as service delivery)?
•    Which of the five Program areas noted above are most and least effective in contributing to REF’s mission objectives?
•    How can REF strengthen its impact?

The external evaluation should cover at minimum REF work in three countries of the Decade of Roma Inclusion: including EU and non-EU countries. There is a particular interest in including one country from each of the following categories: countries where REF directly implements projects, countries in the Western Balkans (as this is the geographic area where REF does most of its work), and EU countries where REF does work but does not directly implement projects.  Country selection should result in an evaluation that accurately reflects REF’s work as a whole. 

The timeframe of activities included in the evaluation should be Jan 1, 2008 to June 1, 2012.

3.    Stakeholders and Intended Audience

The primary audience for this evaluation comprises the REF Board, REF senior management, REF’s long-standing donors, and potential REF donors. 

 4.    Approach, Methodology, and Timeline

In developing the evaluation framework, evaluator(s) will be required to familiarize themselves with and take into serious consideration the agreed-upon results frameworks and preferred evaluation guidelines of the following REF donors, among others:
•    The World Bank, e.g.:
–    “Major evaluation criteria and suggested evaluation questions for DGF funded partnership evaluations” as developed by the World Bank Independent Evaluation Group (IEG) and provided in their Guidance on Program Evaluation
•    Swedish International Development Cooperation Agency (SIDA), e.g.:
–    REF’s results framework as developed for SIDA-funded projects
–    “Glossary of Key Terms in Evaluation and Results Based Management” by the Network on Development Evaluation, a subsidiary body of the Development Assistance Committee (DAC) at the OECD

The selected consultant/group of consultants will be expected to complete the following tasks:

(1)    Elaborate the evaluation methodology and the research instruments, including a desk research template, frameworks for secondary data analysis, questionnaires, and interview/focus group guidelines for primary field research.
(2)    Review relevant organizational documents including but not limited to: annual reports; grantee reports; institutional, external, country-specific, and program-specific evaluations; Country Assessments and updates; research, studies, and policy papers produced or sponsored by REF; and templates for specific programs (e.g. application forms, project implementation plans, quarterly reports, and final reports).
(3)    Collect primary data (through field research) targeting main REF stakeholders (non-governmental and governmental, REF Board members, staff from REF headquarters and the Romanian branch, REF grantees and co-implementing organizations, and student beneficiaries and their parents) in at least three countries.

It is expected that the duration of the evaluation work fit within 40 working days and should occur within the period June–September 2012. The evaluator(s) will be expected to submit the draft report to a review committee (including REF Board members and delegates of the donors) for initial review, revising the draft report according to the comments received and within 15 days of their receipt.
 

5.    Deliverables

The working language of the evaluation is English, although the ability to conduct interviews and research in one or more of the target countries will be helpful. All the deliverables listed below should be provided in English (except the questionnaire and interview guides, which should be translated into local languages as well).
 
•    Notes/ transcripts based on individual interviews and/or focus groups
•    Questionnaire survey database
•    Draft evaluation report
•    Final evaluation report
•    Presentation to the REF board

The evaluator(s) will be expected to present the findings of the evaluation to the REF Board in a draft evaluation report of approximately 35,000 words, including a 2,500-word summary, plus annexes in English (using REF’s style manual).

6.    Required Qualifications and Competences

The selected evaluator(s) are expected to demonstrate the following qualifications and competences:
•    Demonstrated experience in impact and institutional evaluation, with experience as a lead researcher in an impact evaluation being an advantage.
•    Expertise in social science research with appropriate academic qualifications.
•    Knowledge of the policy and institutional framework for the education of Roma in the Decade of Roma Inclusion.
•    Experience working on social inclusion and/or education issues, with experience working with Roma being an advantage.
•    Work experience in South East Europe and especially in one or more of the countries in REF’s focus.
•    Strong comprehension, verbal, and writing capacity in academic English and any other languages of the Decade countries.

In the selection process, evaluator(s) who have not benefitted from REF in the past and are not expected to benefit from REF in the future will have preference.
 

7.    Administrative Oversight and Resources Available

The governing board of REF will provide ultimate oversight for the evaluation. A review committee consisting of Board and donor representatives will review a preliminary draft of the evaluation report and provide feedback to the evaluators as noted above.

The contract with the selected evaluator(s) will include a probationary period; the progression of the contract from this stage will be dependent upon approval of the final work plan, timeline, methodology, instruments, and budget following their calibration after consultation with the REF Board, staff, and funders.

The evaluator(s) should actively consult with REF donors, REF Board members, and staff on the methodology and draft evaluation report. 

REF will assist in organizing field visits and contacts with stakeholders and will cover the costs of these visits as part of the consultancy contract. The evaluator(s) will be provided with reading materials, documents, and all necessary background information by project implementers and REF staff. Most of the required secondary research documents are available on REF’s website; upon request, REF will provide additional relevant documents or samples of them to interested candidates. The evaluator(s) will provide her/his own office equipment and supplies, but will be provided with office space to work in REF offices where available for the purpose of conducting research.

8.    Application Process
Please send a cover letter explaining the consultant’s relevant experience for carrying out the evaluation (maximum 2 page); CV(s) of evaluator/evaluation team, with a description of their roles in the evaluation design narrative.
•     Writing sample(s) in English (preferably a published external evaluation);
•    a proposed work plan with timeline, evaluation outline, appropriate evaluation methodology and the research instruments to complete the assigned requirements (including the desk research template, frameworks for secondary data analysis, questionnaires, timing for the main phases of implementation and interview/ focus group guidelines for primary field research)An itemized budget proposal, including fees and estimated expenses (e.g. travel, accommodation, interpretation, etc.)  for realizing the evaluation;

All listed documents should be sent to the Chair of the REF Board at the following address: cbercus@romaeducationfund.org.
Please mention the name of the position (External Evaluator-Roma Education Fund Impact Evaluation) in the subject line.

The deadline for applications is 22 May, 2012.

Timeline of procedures:
Expected date of contract signing: 20 June, 2012
Draft evaluation to be submitted: 24 September, 2012
Final report to be submitted: 26 October, 2012