Scottish Government

Scottish Compact Baseline Review

CHAPTER SEVEN: MONITORING AND EVALUATION INDICATORS AND BASELINE FORMULATION

7.1 Chapter 4 pulled together work from a variety of sources and identified 11 key factors that any monitoring and evaluation framework would need to use (Paragraph 4.20). Of these, 4 were specific to the Scottish Compact, dealing with the Good Practice Guides and associated guidance. The remainder were generic, dealing with issues such as the Awareness and Trust that seem to be fundamental to any Compact. The first part of this Chapter looks at the extent to which the data sources analysed in Chapter 5 could be used, either in their current form or in some modified way, to develop monitoring and evaluation indicators and baseline information.

DEVELOPING INDICATORS

7.2 There are 2 data sources that seem to have monitoring and evaluation potential. These are:-

• The Direct Funding Database; and
• The Consultation Registration and Evaluation System.

7.3 Table 7.1 identifies the indicators that could be derived from the Direct Funding Database. These are divided into those that could be derived directly, with a minimum of additional work, and the indirect ones that would need some manipulation before they could be formulated. They are as follows:-

• A direct surrogate indicator for Partnership is the amount of funding allocated to particular groups. However this is seen as being an Evaluation, rather than a Monitoring indicator. For example, if over the period of evaluation there had been a policy decision to provide additional support to particular groups or areas (for example young people or rural areas) then the database could be used to track the pattern and amount of spending;
• One of the main factors underlying the effectiveness of the Compact is the development of mutual Trust. One surrogate indicator for Trust is the amount of the Executive's budget that is allocated to the sector. If this increased over time then this could be interpreted as being indicative of the Executive trusting the sector to take on additional roles;
• An indirect indicator is the percentage of funding applications determined within the 3 month period. This is clearly an important issue, as Paragraph 4.14 identified, accordingly efforts should be made to monitor it under the Funding heading;

Mutual Understanding could be monitored by deriving a surrogate indicator: the percentage of applications approved. The assumption is that the greater the percentage of applications that are approved then the more this reflects the sector submitting applications that meet funding rules, thereby showing that, amongst other things, there is greater understanding of the Executive's funding programmes and criteria. Clearly other assumptions could be made. For example,

Table 7.1 Indicators Derived From The Direct Funding Database

an increased approval rate might reflect more resources being made available. However, it seems valid to assume that the Executive applies its funding criteria consistently. Accordingly, even with these different interpretations, it is felt this indicator is valid; and

• Confidence could be measured by 2 surrogate indicators: the percentage of allocations determined within 3 months and the percentage of approved applications with which no problems (such as failure to claim grant payments) are experienced. If the percentage of both indicators increases then it could be seen as indicative of a growth in confidence with both parties having faith that the other will meet its contractual commitments

It will be seen that there is some overlap between the indicators, with one being used to measure both Funding and Confidence. This is inevitable and indeed it would be possible to allocate some of the indicators to other categories given that they are by no means watertight.

7.4 Table 7.2 considers the indicators that can be derived from the Consultation Registration and Evaluation System. It will be seen that more of these can be derived directly than from the Direct Funding Database. However, as with Table 7.1, there is a degree of overlap between indicators. The indicators are:-

• Measuring the time given for receiving, and the time given for reporting on, consultations can be used to monitor conformity to the Consultation Good Practice Guide. One of the key measures would be that at least 3 months are allowed for consultees to submit responses;
• Policy Proofing could be monitored in 2 ways: either directly by analysing the text responses to the Evaluation form and identifying responses that indicate that policy has been changed as a result of feedback from the sector; or by adding to the Form a specific question that asks for examples of any changes that have been made as a result of consultation with the sector (see Paragraph 5.11);
• A surrogate indicator for Partnership could be the extent to which, as part of the consultation exercise, there was liaison with the sector. To this end it might be useful to modify the Evaluation Form so that some of the key sector umbrella bodies (for example SCVO) are identified by name under the "External Organisations" heading;
• Mutual Understanding could be measured by a surrogate indicator, the percentage of total consultation responses that originated from the sector. The assumption is that, as understanding grows, then the sector becomes more willing to respond to consultations. However such an indicator would have to be interpreted with caution as different consultations will clearly have varied relevance to the sector; and
• Confidence can be indicated by 2 surrogates: the percentage of total responses that originated from the sector; and responses from the sector as a percentage of the number of invitations to respond sent to the sector.

7.5 From Tables 7.1 and 7.2 it will be seen that of the 11 areas identified for which monitoring/evaluation indicators need to be derived, the databases could be used to formulate 9 directly and 4 indirectly. However many of the indicators are surrogates: that is they are not directly measuring aspects of the Compact but are measuring things that are tangential and need to be made relevant by making a number of assumptions. Given this there could be debate about their usefulness.

Table 7.2 Indicators Derived From The Consultation Registration And Evaluation System

7.6 It is likely that some of the indirect indicators, derived from the Direct Funding Database, will require additional work by the Executive to devise them. However if this information is not already readily available it seems important that it be produced. Regardless of its value for Compact monitoring, it would seem to be basic management information that any organisation should collate.

DEVELOPING A MONITORING AND EVALUATION SYSTEM

7.7 It appears that 4 of the monitoring/evaluation areas (Awareness, Dissemination, Use and Usefulness) are not covered by any of the data sources. Reference to Paragraph 4.20 shows that the type of indicators needed to monitor these 4 areas are:-

• Awareness could be monitored by finding out if copies of the Compact and the Guides had been received and if staff in government and the sector were aware of them and their contents;
• Dissemination considers the extent to which the Compact and associated documentation has permeated into the organisation. One way of measuring this would be to interview staff at various levels within an organisation to see if they were familiar with the Compact. This is the form of appraisal used in Investors in People accreditation. Such an approach would, however, have considerable resource implications;
• Use would cover the extent to which staff in government and the sector make use of the Compact and Guides in their mutual interactions. Indicators might be the extent to which the documentation is referred to and quoted; and
• Usefulness covers the extent to which the use of the documentation, especially the Guides, was felt to facilitate sector-government interactions. An obvious aspect of this, that might influence formative evaluation, would be to ask for suggestions as to how the guidance could be improved.

7.8 The type of information identified in Paragraph 7.7 is not currently available from any of the datasets. Accordingly it would need to be collected through surveys. It would, however, make little sense to collect such a limited amount of information. There would seem to be more merit in using a survey to collect information on all 11 areas of the Compact as outlined in Paragraph 4.20. The information collected, or derived, from the datasets could then be used to supplement this and add additional dimensions.

THE MONITORING SURVEY

7.9 Members of the Compact Review Group were circulated with a draft survey based on the 11 areas identified as being the essence of the Compact. The survey was designed to be distributed and analysed electronically, with the survey form being sent as an email attachment. Collating the responses electronically means that the questions would need to be closed, with respondents being provided with a range of options from which they would select one. The information gathered would, therefore, be both limited in detail and impressionistic. This approach would, however, minimise costs whilst increasing the response rates. It was also intended that essentially identical surveys would be distributed to the sector and government, thereby avoid the difficulties with the 2000-01 Annual Review which asked the 2 groups different questions. The survey is to be undertaken annually. Although some respondents saw this as being too frequent, on balance it is felt that this is appropriate. It will enable issues to be identified, and appropriate action taken, at an early stage. A longer time interval may mean that concerns are only identified once they have become significant problems.

7.10 There was general agreement that the survey covered the key areas and no additions were suggested. There were a number of suggestions for improvements to the content:-

• That the options "unaware" or "unknown" be added to the multiple choices. It was felt that this addition would avoid a large number of non-responses;
• The wording on some of the questions be revised to make it less ambiguous; and
• Some open ended questions be added so that more of the flavour of the sector-government relationship be picked up.

7.11 In line with the first 2 of these suggestions the survey has been revised. It is included as Appendix 2. It would be easy to add some open ended questions. However there are resource implications, in that such questions would need to be analysed and classified manually. Accordingly it is not proposed that this be done. Instead, to obtain more detailed information about Compact implementation, it is proposed that 2 other complementary approaches be used: focus groups and an Annual Meeting (see Paragraph 7.19).

7.12 There was some concern about the proposal to distribute the survey electronically as it was felt that not all groups within the sector would have access to email. However a number of respondents gave estimates of the percentage of groups that they felt had email access. These were generally high. For example it was estimated that 95% of registered charities had access, whilst a survey commissioned in 2001 by the Executive's Rural Policy Team found that 50% of respondents (many of whom were from the voluntary sector) had internet access at work. Many others had access through community facilities or at home. The only low internet penetration was in small "one-off" community organisations of which only 10% were estimated to have access. Despite these variations, it is still felt that an email survey is the most cost effective approach. Any under-representation of specific groups will be covered through focus group membership and through the sample methodology, which is considered next.

7.13 Chapter 6 considered how to define a sample framework. The conclusions reached were that:-

• Any survey of the voluntary sector should be based on a sample, with the key criteria being that the sample be representative of the sector; and
• The whole of government be surveyed, regardless of the intensity of contact with the sector and the knowledge of the Compact that individual organisations or Divisions might have.

7.14 There are many approaches that could be developed for sampling a representative cross-section of the sector. However, there seems little merit in reinventing the wheel. Accordingly the suggestion is that the Executive work with SCVO and makes use of its Workforce Panel, which is in the process of being set up (Paragraph 6.8). Not only is this being constructed so that it is representative of the sector (in so far as this is known) but, as there will be perceived benefits in participating, the response rates to any survey should be high. The Panel would therefore be used as the sample base for sending out the annual monitoring survey. This would be facilitated by the fact that all Panel members would have email access.

7.15 It is assumed that contact with government will be relatively easily undertaken, given that the Executive carried out the Annual Review. Accordingly it is proposed that the same database be used. It is recognised that there might be problems in gaining a response in some parts of government, in that the survey might not be directed to the appropriate person, whilst some agencies might know little about the Compact and may have done nothing about its implementation. Accordingly it is proposed, in parallel with the survey, that a number of other actions be taken to publicise and implement the Compact within government. These are outlined in Paragraphs 8.4 to 8.6.

SUMMARISING THE CORE INDICATORS

7.16 The original assumption, when the Review started, was that it would be possible to make use of existing data to identify a number of core monitoring and evaluation indicators. As the earlier Paragraphs make clear, this has been only partially possible. Accordingly there is a need to formulate Compact specific indicators and collect these through a survey to supplement those derived from the Direct Funding Database and the Consultation Registration and Evaluation System (Tables 7.1 and 7.2). Appendix 2 gives the proposed contents of the survey.

7.17 In total 34 indicators have been derived from the 2 data sources and the survey. These are summarised in Tables 7.3 to 7.5. For each, the key area it relates to is shown as is its source and frequency of collection. The detailed definitions of each are given in: Paragraph 7.3 for indicators derived from the Direct Funding Database; Paragraph 7.4 for those derived from the Consultation Registration and Evaluation System; and Appendix 2 for the ones that are to be derived from the survey.

7.18 It is proposed that information on each should be collected annually and that the Executive (presumably the Voluntary Issues Unit) should be responsible for collection, analysis and dissemination. As the proposal is that the survey is undertaken electronically, data management is not seen as a problem. Distribution and the collation of results should be easily undertaken once the contact addresses have been assembled.

FOCUS GROUPS AND AN ANNUAL MEETING

7.19 Monitoring the Compact using the datasets and the survey will miss some of the "flavour" of the Compact. Accordingly it is proposed that these sources of information be supplemented by 2 others, designed to gather information in a more discursive way. These are:-

Table 7.3 The Core Monitoring Indicators For Awareness, Dissemination, Use, And Usefulness

Table 7.4 The Core Monitoring Indicators For Consultation, Policy Proofing, And Partnership

Table 7.5 The Core Monitoring Indicators For Funding, Mutual Understanding, Confidence And Trust

Focus Groups, with every 2 years 3 focus groups being convened, covering rural, urban and semi-rural/small town Scotland. Alternatively the groups could be structured along thematic lines, for example covering different client groups. The groups would be used to explore in greater depth issues arising from Compact development and implementation. Each group would be facilitated by an experienced external consultant who would also be responsible for producing a report with recommendations. Membership would be by invitation, with members selected from local voluntary groups operating within the area and the government sector. Attempts would also be made to correct any bias in survey responses by inviting underrepresented groups. Meeting every 2 years is felt to be a reasonable balance between the costs involved in convening the groups and their utility, as more frequent meetings may, after the first, result in few new issues emerging; and

• An Annual Meeting, which brings together sector representatives and government ministers, officials and key staff from NDPBs and agencies, is also proposed. This would discuss Compact implementation, changes needed to make it more effective and other issues that were impacting upon government and the sector. The "model" would be the meetings held in England and Wales. This would be a clear indication that the Compact was given high level political priority and would raise its profile within the sector and government. It may be that existing meetings could be used for this purpose, with their focus changed so that the Compact becomes the umbrella for sector-government discussions.

AN ANNUAL REVIEW AND ACTION PLAN

7.20 The results of the various monitoring initiatives outlined above need to be brought together into an Annual Review. This would summarise the information collected, identify the issues, and discuss how these should be resolved. The Review would be prepared by the Executive and the sector (ideally SCVO) and presented and discussed at the Annual Meeting, at which it would form the "spine" of the agenda. Following any revisions, it would be published jointly by the Executive and the sector.

7.21 Any monitoring exercise, if it is to have a purpose beyond tokenism, has to be part of a process whereby policy is informed and modified. An effective monitoring system has to have a feedback loop, something that did not seem to exist for the 2000-01 Annual Review. To create such a loop it is proposed that there be an Annual Compact Action Plan, along the lines of that prepared in England. This would be the output of the Annual Meeting and the Review. It would identify the specific actions that needed to be undertaken to aid Compact implementation and allocate responsibility for these along with a timetable. Once the first Plan was prepared, then its implementation would form part of the annual monitoring round. Progress would be considered at the Annual Meeting and revisions and changes made which would then be incorporated into the next year's Plan.

DEVELOPING A BASELINE

7.22 Given that the Compact was published in 1998, any Baseline produced in 2003 will be very misleading, in that attitudes measured are likely to have been affected by the implementation of the Compact. Despite this there would seem to be 2 main options for developing a Baseline:-

• To use the results of the first monitoring survey (Paragraph 7.9 to 7.15) as the de facto Baseline and then measure subsequent changes relative to this; or
• To develop a Baseline from the 2000-01 Annual Review (Paragraphs 5.16 to 5.18) supplemented by information from 3 other sources (the Direct Funding Database, the Mapping Exercise and SCVO's Research Panel).

7.23 The Annual Review, although not without its problems, contains a variety of information:-

• For the Executive's Divisions it covers: the main focus of interaction with the voluntary sector; positive and negative experiences; difficulties experienced in meeting Compact commitments; and views on the Compact's usefulness. Responses are summarised quantitatively so that changes can be tracked over time;
• NDPBs' and Agencies' information is similar, covering: the main focus of interactions; positive and negative experiences; difficulties in working with the sector and the usefulness of the Compact. Again the Review gives a quantitative summary that could be tracked over time; and
• For the Voluntary Sector, perceptions of changes in working relationships over the main Good Practice areas are summarised as percentages on a 3 point scale. The Review also gives percentages for respondents knowing about the Compact and citing it in their dealings with the Executive.

7.24 The 3 other sources mentioned in Paragraph 7.22 could be used to provide what may be best described as a contextual Baseline, that is a Baseline that is not Compact specific but which sets the scene within which the Compact operates. These sources are:-

• The Direct Funding Database which should be capable of ready manipulation to provide a variety of information about grant sources and the Policy Priority Area being supported (as outlined in Paragraph 5.3);
• The Executive's Mapping Exercise which provides a partial overview of the Executive's contacts with the sector (Paragraph 5.19); and
• SCVO's Research Panel which gives an overview of the shape and size of the voluntary and community sectors in Scotland (Paragraph 6.3).

7.25 It needs to be stressed that these sources do not provide Compact specific information. However one of the attractions of using them to produce a Baseline is that they exist. Despite this, their use for monitoring and evaluation purposes would mean that the information would have to be updated. There would be resource implications in doing this for the Review and the Mapping Exercise. It is also the case that the former will have been supplanted by the email survey. Accordingly it is proposed that the Baseline be derived from the responses to the first annual email questionnaire (Paragraphs 7.9 to 7.15), supplemented by the indicators shown in Tables 7.1 and 7.2 and additional contextual information derived from the Direct Funding Database and SCVO's ongoing analysis of the sector's shape. The other information sources outlined above could then be used to supplement these as necessary.

DEVISING AN EVALUATION FRAMEWORK

7.26 The monitoring approach outlined above is essentially a mixture of formative evaluation (in so far as the results of the monitoring exercise inform and influence Compact implementation) and summative evaluation, in that some elements of the approach are looking at the outcomes of Compact implementation (for example the growth in Trust and Confidence between the 2 parties). Monitoring and evaluation are therefore not being treated as totally separate processes.

7.27 The outputs of the Annual Review will thus be a mixture of operational issues and more in-depth analyses of the impact of the Compact in terms of developing a deeper, and more fruitful, relationship between the sector and government. Evaluation, therefore, is best seen as being a deepening of the monitoring process. The Annual Review will provide the context. It is then suggested that evaluation be undertaken in the following way:-

• A survey be carried out of government and the sector to gather contextual information on the processes and structures set up to implement the Compact and perceptions of its effectiveness in influencing relationships between the 2 parties;
• A sample of government departments, agencies and NDPBs is selected and the extent to which they are implementing, and have implemented, the Compact be analysed. This will be done by undertaking interviews with staff to assess their awareness of the Compact and to see if it has influenced their interactions with the sector;
• Undertaking a sample survey of voluntary and community groups. The sample would be selected so that it is representative of the sector. Again the aim would be to assess familiarity with, and use of, the Compact, and the extent to which it was felt to have been useful in facilitating interactions;
• The 2 sample surveys would be used to generate a number of case studies, of both good and bad practices in Compact use. These would then be explored in greater depth through interviews with the parties involved in order to identify key issues; and
• A series of recommendations would be formulated intended to make the Compact and its implementation more effective.

The samples (drawn from government and the sector) would be derived from the sample frame used for the Annual Monitoring Survey: that is all government departments, agencies and NDPBs and a representative sample of the sector as included in SCVO's Workforce Panel. There would therefore be a close relationship, in terms of the sample frame, between ongoing monitoring and the less frequent evaluation. Given that there will be an Annual Survey and a series of Focus Groups every 2 years it is felt that, initially, evaluation should be undertaken every 3 years. This is a sufficiently short timescale to allow the results, especially, good practice identification, to influence Compact implementation. As the Compact becomes more mainstreamed then it is recommended that evaluation only takes place every 5 years.

7.28 Table 7.6 outlines in more detail the main areas that the evaluation indicators would seek to explore. The survey would set the context, gathering less detailed information about systems, processes and impacts using a closed questionnaire. This would then be deepened by carrying out semi-structured interviews with staff in government and the sector.

Table 7.6 Key Evaluation Indicators

7.29 This evaluation proposal is not the type of detailed framework that the original tender proposed. This is now not thought to be necessary, or appropriate, given that the suggested approach to monitoring contains elements of both formative and summative evaluation and the fact that the type of indicators that it was originally felt could be generated from existing datasets do not exist.

CONCLUSION

7.30 This Chapter has identified a number of indicators that could be used to monitor and evaluate the Compact. It would be possible to formulate many more. The problem is that to then collect information about them would be time consuming and expensive. Accordingly the suggested approach is one that seeks to minimise costs, whilst covering what are felt to be the main elements of the Compact process. Whilst the use of a broad brush survey undoubtedly has deficiencies, its augmentation by other ways of collecting information, such as the use of focus groups, should mean that the true nature of sector-government relations are identified.

7.31 The use of a monitoring framework blending formative and summative evaluation is felt to be a way of increasing the impact of evaluation.. Rather than the usual "after-the-event" evaluation, which often has limited effect, the suggested approach, especially the link to an Annual Meeting and Action Plan, should ensure that there can be ongoing modifications to the Compact, thereby increasing its effectiveness.

7.32 This Chapter has contained a number of detailed recommendations. The next Chapter repeats some of these by gathering together all of the recommendations of the Review.