Scottish Government

Scottish Compact Baseline Review

CHAPTER FIVE: MONITORING AND EVALUATION INFORMATION

INTRODUCTION

5.1 Having looked at some of the issues surrounding monitoring and evaluation of the Compact, and identified the key areas which any monitoring and evaluation framework needs to focus upon, this Chapter looks at the various information sources that might have the potential to be used for monitoring and evaluation purposes. The Chapter is structured as follows:-

• First 2 continuing and existing data sources, both of which are collated by the Executive, are examined;
• The Consultation Registration and Evaluation System that the Executive is setting up is then considered;
• Two "one-off" information sources are examined; and
• Conclusions are then drawn.

CONTINUING EXISTING INFORMATION SOURCES

5.2 There are 2 information sources that have been categorised as continuing and existing: that is the data is currently collected and collated, has been for a while and will be in the future. The sources are the Executive's Direct Funding and Indirect Funding Databases (Table 5.1). As their titles imply both relate to funding, in particular grant allocations from the Executive and other public bodies. There is already general criticism of compacts for being too dominated by funding issues (see Craig, et al, 1999). Indeed funding is only one of 4 topics covered by the Executive's Good Practice guidance notes. Accordingly there must be concern that, were too much emphasis placed upon funding as a Baseline or monitoring indicator, then the other issues that the Compact covers would be lost sight of and groups not in receipt of funding might be ignored. There are also, apparently, concerns as to how comprehensive the databases actually are. This needs to be borne in mind when considering the information they contain.

5.3 The Direct Funding Database contains details of the voluntary organisations funded by each Executive Department and Division. The information covers:-

• Organisation funded, although it is not always possible from this to know what types of activity the organisation is involved in (for example what does BACUP do?);
• The Department and Division providing the funding;
• The type of grant provided. This is, in effect, the budget heading from which the grant is paid, for example the Millennium Volunteers Fund, Councils for Voluntary Service, or Rural Challenge;
• The legislative basis for giving the funding;
• The Policy Priority Area to which the grant is contributing, for example Rural, Volunteering Sector, Special Educational Needs or Older People; and
• The amount allocated in the relevant financial year.

Table 5.1 Continuing Existing Information Sources

5.4 As this information is on an Excel spread sheet it should be easily manipulated if there were a desire to aggregate data by Policy Priority Areas or Type of Grant. Time series information is also available as the dataset goes back 4 years. It should, therefore, be possible to look at trends in funding by the various categories listed above.

5.5 The Indirect Funding Database has similar details of voluntary organisations funded by Departmental sponsored Non-Departmental Public Bodies (NDPBs) and agencies, going back for 3 years. The information includes:-

• The Department or Division providing the funding;
• The NDPB or agency responsible for managing funding;
• The voluntary organisation funded, for example Social Firms Scotland and Young Scot Enterprise. However, as with the Direct Funding Database, the activity being supported is not always obvious;
• The amount of planned spending; and
• A regional breakdown, albeit that this excludes the distribution of resources under a number of headings (see Paragraph 5.6).

Again, as the information is contained on a spread sheet, various aggregations should be possible.

5.6 Unfortunately some of the very large funding allocations are not disaggregated to specific end users. For example, in 2001/2002 out of total funding of 304 million, 177 million was distributed to various unspecified housing associations, 48 million to "multi-agency partnerships pursuing Social Inclusion Strategies" whilst a further 34.8 million was distributed by the Scottish Arts Council to bodies described as "various". In total, therefore, 85% of funding was allocated through 3 out of 721 funding categories for which no breakdown by area or activity is given.

5.7 The limited disaggregation of the Indirect Funding Database therefore means that its use for monitoring and evaluation purposes is limited. The Direct Database may have more potential. However its financial focus means that it is only covering one of the Compact's key areas, whilst the tangential relationship of the information to the Compact may mean that it can only be used to provide surrogate variables.

CONTINUING FUTURE INFORMATION SOURCES

5.8 The Executive is in the process of setting up a Consultation Registration and Evaluation System, which incorporates a Registration and an Evaluation Form. This is to become operational in April 2003 and will eventually provide continuing, or time series, data (Table 5.2). As both forms cover policy consultation they overlap with a key area of the Compact.

Table 5.2 Continuing Future Information Sources

5.9 The Consultation Registration Form will hold details of forthcoming consultation exercises and the information will be entered into a database. This will include such data as the target audience, consultation methods and timetables. Information on the target audiences identifies "Voluntary Sector: Umbrella Groups", "Voluntary Sector: Community Groups" and "Equality Groups", with Equality Groups being further subdivided into 12 interest groups, for example Older People and those on Low Incomes. Many of these are either of interest to the voluntary sector or will have voluntary sector representative bodies.

5.10 The Consultation Evaluation Form, to be completed after a consultation has closed, covers such things as the impact of the consultation, liaison, feedback given to consultees and the time given for responses. It will be possible to relate the evaluation responses back to the target groups identified in the Registration Form. Accordingly, it should be possible to use the Evaluation data to monitor such things as the time allowed to receive consultation responses and the feedback being provided to consultees.

5.11 It might also be possible to look at the impact of consultation on policy, thereby enabling aspects of the Compact, such as proofing, to be evaluated. However the information needed to do this will be provided as text, with respondents being asked to give a summary, of not more than 150 words, on what has been done as a result of consultation. There will, therefore, be resource implications if it were to be analysed. It might, however, be worth considering if the Form could include a more specific question, addressed at proofing. For example, something along the lines of, "Please outline any changes made to policy as a result of consultations with the voluntary sector", would give some indication that the impact of policy on the sector was being taken account of.

5.12 When the System is implemented, and if it has widespread support across the Executive, it would seem to have considerable monitoring and formative evaluation potential. Completion of the Consultation Registration Form is to be a requirement if consultation teams are to have their consultation paper published on the Executive's website. This sanction may mean that compliance is high. However there do not seem to be any sanctions if the Evaluation Form is not completed, although it is the intention to send an email reminder to all who have registered a consultation. Despite this, the variable response to other data collection exercises within the Executive (see Paragraph 5.19), may mean that compliance may not be 100%.

5.13 The data collected would also have to be disaggregated in so far as the voluntary sector accounts for only 2 out of the 12 target audiences identified. However, as the data will be captured, and held, electronically, this should not be a major problem.

5.14 Greater difficulties may be caused if there is a lack of understanding amongst respondents as to the nature of the voluntary sector. The responses to an earlier Mapping Exercise, undertaken by the Executive, demonstrated a degree of confusion amongst some as to how voluntary groups were defined (see Paragraph 5.21). If this is widespread within the Executive then the information collected through the System may be of limited use.

"ONE-OFF" INFORMATION SOURCES

5.15 The Executive has undertaken 2 data collection exercises (one in conjunction with SCVO) that have potential to be used to review, or form a Baseline, for the Compact (see Table 5.3). The first of these was Compact specific.

5.16 The Annual Review of Implementation of the Scottish Compact was based on returns to 3 questionnaires. These were sent to:-

• Scottish Executive Divisions;
• NDPB and Agencies; and
• 400 voluntary and community bodies on SCVO's Research Panel (see Paragraph 6.3). This latter survey was undertaken by SCVO.

5.17 Although the Review is the main source of Compact specific information it is not without its problems. The main ones are:-

• The variable response rates. Given this, some of the impressive percentages quoted in the Review are based on small numbers of responses;
• The differences in the questionnaires, with the Divisional, NDPB and agency questionnaires consisting largely of open ended questions dealing with such things as experiences of working with the voluntary sector and difficulties encountered in trying to meet the Compact's commitments. In contrast the voluntary sector questionnaire mainly asked respondents to rate their experiences, of such things as partnership working and funding issues, on a 3 point scale. This makes comparisons difficult as partners to the same agreement were being asked different things;
• Implicit in the questionnaires, certainly the voluntary sector one, was the view that the Compact was about government changing its ways and attitudes towards the voluntary sector. Thus the voluntary sector questionnaire was essentially an assessment of the sector's perceptions of how government has changed. The governmental questionnaires were more balanced, in so far as questions were asked about the implementation of the Compact internally, through such mechanisms as objective and target setting. These differences, although a good reflection of much of the writing on compacts (which tends to imply that compacts are about government making commitments and changes rather than the voluntary sector), and also a fair reflection of the one-sided nature of the Good Practice Guides, mean that bringing the two sets of results together, in an way that reflects the concept of the Compact as a partnership, is difficult; and
• The focus of all questionnaires was on process and, to a lesser extent, outputs rather than outcomes or impacts. However this was, perhaps, inevitable given that the Compact was still being established and had, as the responses show, yet to permeate very deeply into the organisations being surveyed.

Table 5.3 "One-Off" Information Sources

5.18 The Review document, unfortunately, does not always summarise the information that was requested through the questionnaires. A particular case in point is the question to the Executive's Divisions on having an objective in their workplans regarding responsibilities for Compact implementation. There is a similar question in the agency and NDPB questionnaire. However, neither set of responses is summarised in the Review. This is disappointing as such information could form a useful input into a Baseline which could then be monitored. One can only assume that the omission of this information was either because respondents did not provide it or that very few had such an objective.

5.19 More recently the Executive undertook a " Mapping of Scottish Executive contact with the voluntary sector", albeit that no analysis has been done, as yet, of the information returned. This was an initial exercise done for internal purposes. As a first attempt at mapping contacts with the sector across the Executive it is perhaps not surprising that there are felt to be some problems with the robustness of the data. The exercise also raised methodological issues regarding the sample framework. For example it was unclear if all Departments or Divisions should be included as the extent of contact with, and influence over, the sector was felt to vary (see Paragraph 4.11). This might then explain the low response rates from some Departments. This is an issue that is looked at in greater detail in Chapter 6.

5.20 Accepting these caveats, the various Divisions were asked to report on their contacts under 4 headings:-

• Published policies and consultations that impact upon the sector;
• Current issues that have implications for the sector;
• Links with intermediary voluntary organisations; and
• Working and review groups that involve the sector.

5.21 A rapid scrutiny ⁸ of the information shows that:-

• Of the 94 Divisions identified, responses were received from over two thirds;
• The respondents identified 82 links with intermediary bodies and 96 links with groups on which the voluntary sector was represented. However there was, not surprisingly, considerable overlap. Many organisations, especially the umbrella groups, had links and involvement across a large number of Departments and Divisions. For example SCVO was mentioned 17 times as a intermediary body (21% of responses) and 8 times as a member of a group (8% of responses);
• The frequency and regularity of meetings with groups varied widely. For example Community Care 1, Branch 4 met with the Mental Health and Well Being Support Group 12 times a year. Others were less structured, as can be seen from such comments as "all voluntary organisations in theory can be part of the partnership process"; and
• Some of the responses indicated that there was a need for wider understanding of the voluntary sector. For example one respondent stated "I am …not clear how voluntary bodies are defined", whilst another said that she did "not really know what intermediary voluntary organisations are". Whilst these responses may indicate a failure to read the guidance given on the form it would seem to be indicative of the types of problem that any survey dealing with the sector-government interface is likely to encounter.

5.22 The 2 "one-off" information sources are therefore not without problems. Particular problems with the Mapping information are the level of non-responses from some Divisions and the variations in the detail provided. The Review suffers from low return rates from some categories of respondents, although any alternative is likely to face similar difficulties.

5.23 Despite these problems, in their current form, both sources could be built upon to provide a Baseline. For example, the Review contains some useful information such as a snapshot of the views of the voluntary sector on their relationships with government over all of the 4 areas of the Compact (Funding, Consultation, Partnership Working and Policy Proofing). The Mapping information provides a snap shot of Executive-voluntary sector networks which could also be fed into a Baseline.

5.24 If these 2 sources are to be used in developing a Baseline then one implication is that they be repeated at regular intervals in order to monitor progress. However any such repetition would need to ensure that some of the problems outlined above were overcome. This might mean modifying and adding questions.

CONCLUSIONS

5.25 Although a variety of information is available, the majority has been collected for non-Compact purposes. Using the unmodified data for monitoring and evaluating the Compact could be difficult as there is often limited correspondence between the data and the Compact's objectives. Its use to provide a Baseline would also seem to have problems. Despite this, the various datasets do have some potential. The use to which the data could be put is dealt with in Chapter 7. The exception is the Indirect Funding database. The extent to which the data is aggregated means that it is hard to see it being of much use.

5.26 Given that the data has limited use it is likely that there will be a need to collect information through surveys of the sector and government. Accordingly the next Chapter considers the issues involved in devising any sampling framework.