Scottish Compact Baseline Review

Listen

Scottish Compact Baseline Review

EXECUTIVE SUMMARY

INTRODUCTION

1. The Baseline Review was commissioned from GEN Consulting to develop recommendations for setting up a monitoring and evaluation framework for the Scottish Compact.

2. The Compact was launched in 1998 and sets out a framework of principles covering the working relationships between central government, Non-Departmental Public Bodies (NDPBs) and its agencies in Scotland and the community and voluntary sectors. These principles are outlined in the formal Compact documents and a series of Good Practice Guides. They aim to develop better mutual understanding and confidence. As such the Compact development process is about relationship building and maintenance.

3. The Review involved a number of elements. These included: a literature review; a comparative analysis of other compacts; an appraisal of the various datasets held by the Executive and others; an assessment of the extent to which the datasets could be used for monitoring and evaluation purposes; and the formulation of a series of monitoring and evaluation recommendations.

COMPARATIVE ANALYSIS

4. The comparative analysis involved looking at the other national compacts in the United Kingdom and the Canadian Compact. From this it proved possible to formulate a generalised model of compact development. The start of the process is invariably discussion between the sector and government out of which comes some commitment to set up a compact. The formal compact is then produced. If this is not produced jointly, by the sector and government, then there is, at least, consultation between the 2 parties. The compact is then implemented through the production of good practice guides, the commonest of which invariably deals with finance.

5. All of the compacts analysed followed this model. Thereafter practices differ. Some, for example the English Compact, produce an annual action plan which details what needs to be done to implement the compact over the next year. This is usually linked to some form of review process, in effect monitoring. Some compacts then link the review into an annual meeting. This often brings together senior politicians (government ministers in the case of the English Compact) and senior representatives of the sector. Progress is discussed, a report of the meeting published and the next year's action plan formulated. The cycle of implementation then continues.

6. All compacts recognise the importance of monitoring and evaluation. Where monitoring has been undertaken, for example in Scotland and England, this has involved surveys of the sector and government. The extent to which the outputs of monitoring (or review as it is often called) feed into the compact implementation process varies. For example, the review undertaken of the Scottish Compact does not seem to have influenced the implementation process. This does, however, happen in England and Wales. All compacts have a commitment to evaluation. None have yet undertaken any, nor do any seem to have firm ideas about either the process or the content of such an evaluation.

7. The compacts were compared on a variety of development and process criteria. The key points are that:-

  • Only 1 compact, that in Northern Ireland, was preceded by a baseline survey which provided a snap shot of relations between the sector and government;
  • Only the Welsh compact has a statutory basis, there being a commitment in the 1998 Government of Wales Act to have a scheme which is to promote the interests of the voluntary sector;
  • 3 out of the 5 compacts are symmetrical, that is they set out the obligations of both the sector and government. The other 2 seem to see the compact as being less a partnership than a set of guidelines to which government is to conform in its dealings with the sector;
  • The numbers of codes of practice vary from 1 to 4, albeit that this comparison may be slightly misleading as the detail in some of the formal compact documents means that they are almost codes of practice;
  • The extent to which the codes are symmetrical, that is outline what is expected of both parties to the partnership, again varies, with only 2 of the compacts being judged to be symmetrical. The Scottish Compact was classed as asymmetrical, albeit that the Scottish Council for Voluntary Organisations (SCVO) had produced guidance outlining the sector's commitments under the Compact. Although this was useful, it was felt that it diluted the Compact's message and potentially fragmented its impact;
  • 3 of the compacts had action plans which broke the implementation process into tangible parts. Often these plans identified who was responsible for implementation and set time scales;
  • Annual reviews and reports were produced for 4 of the 5 compacts, albeit that in some instances, such as Scotland, these did not seem to be linked into the implementation process; and
  • Finally a judgement was made as to the extent to which there was involvement of elected politicians in the implementation process. In Wales and England this was classed as "high" in that senior politicians had major involvement in the process, often chairing annual meetings to review progress. In Scotland involvement was felt to be low. This is not to suggest that there is not contact between politicians and the sector, but that there seems to be little contact under the Compact "badge".

THE AREAS FOR MONITORING AND EVALUATION

8. A variety of information was synthesised to identify the main areas that monitoring and evaluation should focus upon. Eight main areas were identified:-

  • Awareness of the compact, which was seen as being the start of any relationship building process;
  • Dissemination, that is the extent to which the knowledge and awareness of the compact permeates through an organisation, especially to the front-line staff;
  • Usage was seen as being a critical measure, as it was likely that there would be limited impact, regardless of the extent to which there was awareness of the compact, if the parties to it made limited use of it;
  • Usefulness then needs to be considered, that is the extent to which the partners find the compact of use in their dealings with the one another;
  • The good practice guides, given that they are often the most tangible and detailed aspect of the compact, then need to be considered in terms of: the extent to which they have been used; and the impact that they have had;
  • Given that the compacts are about relationship building Mutual Understanding then needs to be considered, that is the extent to which the partners have developed a better understanding and awareness of one another;
  • Confidence, (the day-to-day functioning of the relationship) needs to be considered; and
  • Finally Trust, that is the ultimate goal of the compact, needs to be assessed, with the development of trust being the culmination of confidence.

It is felt that the above cover all of the various ways in which the compact can be defined, that is: as state of mind; the formal document; the good practice guides; and, implicitly, through the supporting documentation.

MONITORING AND EVALUATION INFORMATION

9. A variety of information sources were evaluated to see if they could be used to provide information for monitoring and evaluation of the Scottish Compact. The majority of the sources had been set up for non-compact specific purposes. As such it was felt that they were of some, although limited, use in an unmodified form. The exception was the Indirect Funding database, which deals with projects funded by Executive sponsored NDPBs and agencies. Over 85% of funding was not disaggregated to specific end users. As such it was felt to be of little use.

10. A variety of indicators were proposed that could be either drawn directly, or derived, from 2 main data sources: the Direct Funding database (which contains details of voluntary organisations directly funded by the Executive) and the soon to be set up Consultation Registration and Evaluation System (which will log information about forthcoming consultations and then summarise the consultation process and its impact).

11. A total of 34 indicators were identified and each was linked to the key areas for monitoring and evaluation, as outlined in Paragraph 8. For some of the indicators to be operationalised additional information needed to be collected whilst others required modifications to the data collection forms. The suggested changes are detailed in the full report.

DEVELOPING A SAMPLING FRAMEWORK

12. Defining the partners to the Compact is easily done at a strategic level. However once one begins to unpick the "voluntary and community sectors", and "government", the situation becomes more complex.

13. Given that the voluntary sector is so large, it being estimated that there are in excess of 20,000 individual organisations, monitoring and evaluation needs to be based on a sample of the sector. To identify a voluntary sector sampling framework a variety of networks were analysed, and the advantages and disadvantages of each outlined. Although the Compact explicitly accepts that not all voluntary organisations will want partnership with government, the view was taken that the entire sector is affected by government. The intensity of this impact will clearly vary and some parts of the sector will be far more enthusiastic about developing close links with government than will others. However it was felt that the Compact is relevant to all voluntary and community organisations. If this is accepted, then the crucial factor for any sampling framework is that it is representative of the sector and does not give undue weight to various sub-groups: for example national organisations or ones funded by the Executive. It was felt that the most cost effective way of developing a sample framework was for the Executive to work with SCVO in order to make use of one of its data collection networks. The suggestion is that this be the Workforce Panel which is currently being set up.

14. The government parties to the Compact are more easily identified. However it can be argued that it would be difficult to identify individuals within large organisations who are aware of the Compact and are involved in its implementation. Whilst this may be the case, it was felt that this reflected failures of internal management and an unwillingness to take on board the commitments contained within the Compact. Accordingly it was recommended that monitoring and evaluation should include all parts of government covered by the Compact, rather than being based on a sample.

RECOMMENDATIONS

15. A number of Recommendations were then formulated. These were seen as being ones that had minimum resource implications and, as far as possible, made use of existing information. In some respects the Recommendations exceed the brief in so far as they deal with issues of process and procedures. However it was felt that the Compact needed to be "sharpened up" and its implementation made less permissive, if it were to be capable of being monitored and evaluated.

16. The key Recommendations for process and procedures are that:-

  • The Compact and associated Good Practice Guides be revised and updated. The revised versions need to be symmetrical, identifying the commitments of both government and the sector;
  • An Annual Compact implementation Action Plan be drawn up;
  • There is an Annual Meeting which brings together the sector, senior politicians and civil servants and senior management from NDPBs and agencies. This would discuss progress in implementing the Action Plan and associated issues. The outcome of the Meeting would feed into the next year's Plan; and
  • Within each government organisation a Compact champion be identified who will have responsibility for ensuring that it is implemented.

17. The Recommendations related to monitoring are that:-

  • A number of direct and indirect monitoring indicators be derived from the Executive's information sources and that additional information be collected that relates specifically to the content of some of the Good Practice Guides;
  • The key monitoring mechanism should be an Annual Monitoring Survey that would be distributed and analysed electronically. As far as possible this would ask identical questions of the sector and government. The survey of the sector should be based upon a sample derived following consultation between the Executive and SCVO. The government survey should be a census;
  • The survey should be augmented by 3 Focus Groups that would be convened every 2 years and would cover different types of geographical area.

18. Collating a Baseline for the Compact is difficult, given the length of time that has passed since it was first introduced. Accordingly the Recommendation is that:-

  • The baseline is based upon the responses to the first electronic survey, supplemented by information derived from the proposed indicators and contextual information collated from other sources.

19. The approach to monitoring proposed blends summative and formative evaluation. Given this, it is felt that formulating a separate evaluation framework was of limited importance. Accordingly it is recommended that:-

  • Evaluation is based upon surveys and interviews which analyse the processes within a sample of voluntary organisations, government departments, agencies and NDPBs. These would be augmented by case studies.

20. Fuller details of the Recommendations and the reasoning behind them are given in the main sections of the report.

Page updated: Wednesday, April 05, 2006