Scottish Government

ANNEX G: SUMMARY OF INDICATORS, TARGETS AND BASELINES

Introduction

At the request of DG Employment, the following non-paper has been drafted to describe the approach to indicators and targets for the 2007-2013 European Social Fund Programmes in Scotland. The non-paper sets out Scotland's main principles in developing its 2007-2013 reporting/evaluation system. The July 2006 paper presented a number of key objectives in developing the new approach, including:

• producing a limited, but more meaningful number of indicators;
• introducing more mandatory reporting on indicators by projects;
• providing clear guidance on reporting, especially indicator definition;
• distinguishing between reporting for information and reporting for targets; and
• using more longitudinal and follow-up surveys to understand the long-term effects of funding better.

These principles underlie the approach set out in the draft OPs. In summarising this approach, this paper covers the following:

• a description of the overall approach to selecting indicators and estimating targets for the Programmes;
• a list of the key research resources used for the quantification work;
• a detailed breakdown of the indicators and targets for each of the priorities;
• a description of the baselines for the different Programmes; and
• arrangements for gathering and monitoring data.

Overall Approach

The approach to indicators and targets in the Scottish OPs has been designed to comply with the relevant regulations and has been fully informed by Commission guidance. These include the following documents:

• Structural Funds Regulation 1083/06 Article 37.1
• ESF Regulation 1081/06 Article 4.4
• Working Paper 2: Indicative Guidelines on Evaluation Methods: Monitoring and Evaluation Indicators
• Working Paper 3: Indicators for monitoring and evaluation: an indicative methodology
• Working Paper 5: Indicative Guidelines on Evaluation Methods: Evaluation during the Programme Period
• Working Paper 6: Measuring Structural Funds Employment Effects

In line with Commission guidance, a 'life-cycle' approach to designating indicators has been used. Under each priority, and for the Programmes as a whole, indicators are identified which distinguish between:

• outputs: as a measurement of initial directly-funded activity;
• results: as a measurement of the interim direct and immediate effects on direct beneficiaries; and
• impacts: the longer-term effects arising from the funded actions.

The choice of indicators also takes account of the need to measure not just gross but net effects of actions as well (ie. taking account of substitution, additionality, displacement and other indirect effects).

In our approach, we have distinguished between indicators on which we will report for information and indicators on which we are setting targets to track Programme performance. For information purposes, there are a number of fields on which the Programmes will gather data and provide regular reports to the Commission. This is being done in line with the reporting obligations of Annex XXIII of the Commission's Implementing Regulation and includes the following information on beneficiaries:

• employment status;
• age range;
• vulnerable groups, including minorities/migrants and disabled individuals; and
• educational attainment.

However, the Programmes will not necessarily set targets for all these indicators. Targets will only be drawn up for outputs and results indicators that directly reflect on the explicit aims and activities of the individual priorities. Measuring performance in these areas will be essential. Impact indicators typically have not had targets set, owing to the significant methodological problems of calculating net effects and counterfactuals where EU funding is relatively small with respect to the size of the regional economy it targets.

While indicators and targets are priority-specific, across both ESF Programmes there are common targets that apply across the priorities and which will be reported on a Programme as well as an individual priority basis:

• Number of participants receiving support
• Number of participants entering employment
• Number of participants gaining a partial (ie. a completed unit towards a full qualification) or full qualification

Background Research

The indicators and targets have been developed on the basis of parallel research exercises. First, the past experience of previous Programmes was examined by analysts in the European Structural Funds Division to guide target-setting for equivalent indicators in the 2007-13 Programmes. In particular, performance on the 2000-06 Objective 3 and the Highlands & Islands Special Transitional Programmes were assessed through the Annual Implementation Reports. These were used to provide realistic estimates for the targets for the 2007-13 period. The mid-term evaluations and mid-term evaluation updates of the Scottish ESF Programmes were also used to assess the robustness of future indicators. Recommendations set out in those evaluation reports on monitoring and evaluation have informed the development of 2007-13 systems and procedures.

Second, key studies of the 2000-06 Programmes were used to determine appropriate targets. A GB-wide beneficiaries survey was commissioned of MORI in 2005 to follow up individual beneficiaries of projects supported under the ESF programmes. For this research, separate surveys were conducted in England, Wales and Scotland, building on similar work conducted in earlier years and carried out by MORI. It provided detailed analysis of impacts on beneficiaries of different activities and priorities. In addition, as part of the mid-term evaluation update in 2005, a specialist study was commissioned of EKOS Consultants to review the effectiveness of social inclusion activities within the Objective 3, Objective 2 West of Scotland and the Highlands & Islands Special Transitional Programmes for 2000-06. Again, the study contained useful information on impacts.

Third, a special study on indicators and targets for the 2007-13 Programmes was commissioned of DTZ. The recommendations of the report - which included specific suggestions for indicators under different priorities - have shaped the choice and definition of indicators in the draft OPs.

Lastly, as targeting progressed, use was made of the 1998 EKOS benchmark study on impacts of different Structural Funds-supported actions. Although conducted some time ago, the study set out useful cost-per-job benchmark comparators against which the performance of the 2000-06 Programmes could be tested and the targets for the 2007-13 Programmes be estimated.

Priority 1

For both 2007-13 ESF Programmes in Scotland, the same priority structure has been employed. While the priorities have different emphases between the Highlands & Islands and Lowlands & Uplands Scotland, the overall themes of each priority - and to an extent, the indicators chosen - are largely common between both Programmes. Hence, the following sections consider indicators within priorities.

Priority 1 targets those outside of the labour market by providing targeted assistance to those groups experiencing the greatest challenges in gaining sustainable employment. It particularly aims to support those who are furthest from the labour market and in need of the most assistance.

Within Lowlands & Uplands Scotland, the priority ('Progressing into Employment') will only apply in parts of the region, principally those Local Authority areas where social deprivation and regeneration challenges are particularly acute. It is the only ESF priority where such geographical targeting is being employed.

The outputs reflect the priority's targeting of key groups experiencing severe problems of worklessness, especially those with multiple deprivations (which will be set out as part of guidance to applicants, but can be defined as those belonging to more than one target group) and those in the so-called NEET Group (young people not in employment, education or training). In this context, the Programme does not distinguish between 'worklessness' and 'unemployment' in targeting, as the experience of the 2000-2006 Programme has shown that it is not a useful distinction from the perspective of individual projects and beneficiaries, although the distinction will be retained in terms of reporting on monitoring data on beneficiaries.

Results have been defined in terms of the key goals of the priority: getting individuals into work (number of participants entering into employment), keeping individuals in work (number of participants in work six months after leaving) and encouraging individuals to gain qualifications and new skills (number of participants entering into education and training and the number gaining a partial or full qualification).

In addition, beneficiary data on outputs and results will be reported for all target groups set out in the priority description, though targets will not be explicitly set for these groups.

Given the focus on the priority on those which are far from the labour market, it is anticipated that the impact of support may not necessarily be immediate employment for participants, but significant progress towards gaining the confidence and skills to gain sustainable employment. As a result, the key impact measures will be an indicator examining progress of beneficiaries towards employment, as well as the final net number of those securing employment. The first is a 'soft' indicator of "distance travelled" towards the labour market which we will be developing over the coming months. Targets have not been set these indicators as more work needs to be done to gather meaningful estimates on which targets can be set.

Several of these indicators which will be measured through specially-commissioned data-gathering exercises. The 'sustainability' of employment will be measured through follow-up surveys on the number of participants who remain in work six months after leaving - data on these beneficiaries surveys will be collected in 2009, 2011 and 2014. Net employment will be measured at the same time.

For Lowlands & Uplands Scotland, the target indicators are as follows.

Priority 2

Priority 2 concentrates on investing in the skills of those in work, but concentrating on particular groups and issues, including: the low-skilled and low-paid who have difficulty finding sustainable employment; tackling the gender pay gap by reducing gender imbalance across different sectors; the skills needs of business managers, especially entrepreneurs; and social enterprise skills needs. Given the distinct sub-themes within the priority, indicators are best considered on this basis.

In terms of addressing the low-skilled and the low-paid, activities will focus on improving the skills of beneficiaries to levels 1, 2 and 3.

The second theme of the priority is to address under-representation in the workforce by encouraging individuals of both genders to enter sectors dominated by either gender. Outputs will be defined in terms of participation in projects addressing gender imbalance, as identified and subsequently tagged by the Managing Authority. Results will reflect the number of participants going into gender-imbalanced sectors, which will be defined by the Managing Authority in advance of the first round of Programming.

The third theme focuses on improvements to the entrepreneurial and business skills of managers. The output would be defined in terms of participation in specific entrepreneurial and business skills projects, as identified and categorised by the Managing Authority. The result would be obtaining partial/full qualifications. The impact would be assessed in the second half of the Programme by examining whether the skills training has benefited companies.

The last theme could be considered a variant of the third - improving the skills of social enterprises. The indicators would reflect the same approach taken under the third theme. Again, the impact would be measured by specially-commissioned survey work.

For Lowlands & Uplands Scotland, the target indicators are as follows.

Priority 3

For Lowlands & Uplands Scotland, the target indicators are as follows.

Priority 3 in the two ESF Programmes focuses on widening access to lifelong learning and training by improving the capacity of lifelong learning organisations and their lifelong learning tools and services. Under this priority, the output indicators cover individuals (number of participants who are trainers) and systems/materials (number of projects supporting the development of new courses/materials or innovative approaches to learning). Results will defined along similar lines: for example, the number of trainers receiving qualifications and number of new courses/materials developed. Ultimately, the impact of the priority will be felt in the number of participants gaining partial or full qualifications as a result of the supported activity - this would be measured in the second half of the Programmes.

Baselines

Article 37 of the General Regulation states that Programme documents should contain "information on the priority axes and their specific targets… the indicators shall make it possible to measure the progress in relation to the baseline situation and the achievement of the targets in the priority axis". In view of the proportionality principle and the requirement to limit the number of indicators to make their monitoring effective, there are not quantified indicators for every target group/activity. Moreover, some data will be collected through individual participants data collection and some through specific follow-up survey/evaluation work. The Managing Authority will set out national definitions for each of the indicators as part of guidance to applicants on monitoring in advance of each programming round. When reporting the information required by Annex XXIII of the Implementing Regulation, EU definitions will be used.

Baselines for measuring progress in achieving progress for each priority are set out below. They have been selected with a view to capturing the key goals of each priority and will be assessed in the second half of the Programme, although attributing changes in these regional indicators to Programme activity may be difficult to make because of the small size of Programme resources relative to the region's economy.

• For Priority 1: the levels of economic activity in the region and working-age people in education, employment or training (the Programme should contribute to the share increasing), and the share of NEET and sick and disabled benefit claimants (the Programme should contribute to these shares decreasing).
• For Priority 2: the share of employees below the low-pay threshold (the Programme should contribute to the share decreasing), the shares of economically-active men and women with different skills levels and female earnings levels (the Programme should contribute to these shares increasing), the new firm formation rate and the gross value added of the region relative to the UK (the Programme should have a positive impact).
• For Priority 3: share of adult participation in education and training (the Programme should contribute to this share increasing).

Arrangements for Monitoring

Monitoring will be the joint responsibility of the Managing Authority, the Monitoring Committee, partners and the Intermediate Administration Body designated by the Managing Authority, in accordance with Articles 63-66 of the General Regulation. These bodies will be responsible for setting up appropriate monitoring systems, defining indicators and disseminating knowledge of their use, ensuring that projects fully understand their reporting obligations, assuring themselves on the quality and coverage of the indicator data, and providing appropriate reporting on the basis of that data.

Information on the different indicators will be collected in two ways. Some indicators will be collected on an on-going basis through individual project data. This information will be provided directly to the Managing Authority by projects on a quarterly basis. Data on other indicators will be gathered through specially-commissioned surveys.

In essence, project partners will be responsible for reporting on the indicators set out under the priority descriptions, as appropriate for their project, on a quarterly basis. These reports will be initially checked for consistency by the Intermediate Administration Body and the information will be stored and analysed on a regular basis by the Managing Authority. Analysis of the information will feed into reports to the Programme Monitoring Committee - twice-yearly for regular project reporting but also on an annual basis for the Annual Implementation Reports discussed below. The data will also be used as part of the thematic evaluations detailed in the previous section.

A quarter-year monitoring regime of projects will ensure the quality and effectiveness of implementation of projects through assessment of progress towards achievement of the indicators defined in the Programme. Management information systems will provide core data on a regular, timely and consistent basis throughout the programming period, enabling the Managing Authority and Programme Monitoring Committee to monitor Programme performance and results in line with their responsibilities. A set of core management information requirements for ESF and co-financed activity will:

• enable progress towards the achievement of financial, output and result indicators to be assessed at priority level;
• provide performance measures for the Managing Authority and the Programme Monitoring Committee;
• feed into preparation of the reporting requirements set out in the next section;
• allow transfer of data to the Commission as required in the Commission's Implementing Regulation, particularly Annex XXIII; and
• contribute to the wider evaluation of the Programme - where necessary, the core information requirement will be supplemented by other information such as follow-up surveys as set out in the section above.

The use of the indicator information for evaluation purposes will be set out in the Scottish Evaluation Plan for all the 2007-2013 Structural Funds Programmes, which will be submitted to the Commission after the approval of the Programmes. This Plan will be overseen by a specially-designated Scottish Monitoring and Evaluation Group, which will be responsible for proposing the thematic evaluations to be conducted on any of the Scottish Programmes.

The following table sets out indicative timescales for known evaluations. The Evaluation Strategy will set out the thematic evaluations for the Programme on an annual basis.

European Structural Funds Division
Scottish Government