Scottish Government

Scottish Compact Baseline Review

CHAPTER SIX: DEVELOPING A COMPACT SAMPLING FRAMEWORK

INTRODUCTION

6.1 At a general level it is easy to determine the parties to the Compact agreement from which any sample should be drawn: government, as defined in the Compact documentation, and the voluntary and community sectors. However the situation becomes more complicated when each of these constituencies is explored in greater detail. Accordingly the purpose of this Chapter is to look at some of the issues involved in identifying the sample frame. This develops in greater depth some of the issues touched upon in Chapter 4.

DEFINING A VOLUNTARY SECTOR FRAMEWORK

6.2 Defining the voluntary and community sectors with any precision is difficult. The main reason for this is that, although various networks and datasets exist, they cover only a part of the sector, for example registered charities. There are, however a number of networks, some of which are already used to collect non-Compact specific information (Table 6.1). These might have the potential to be used in a way that could either inform a compact Baseline or could form a sample framework for collecting monitoring information on a systematic basis, either using existing data as surrogate variables or by collecting additional information. All of these networks are facilitated or convened by SCVO.

6.3 The first is SCVO's Research Panel of Voluntary Sector Organisations. This is a stratified sample of 5,000 voluntary organisations. It now holds data going back 10 years. Its stratified sample basis means that it is possible to gross up statistics and obtain data that is reasonably representative of the sector as a whole. The data held covers:-

• A breakdown by "field of work". This might have potential to be used for measuring specific interests that government has. For example if the Executive decided that "services for elderly people" were to be a political priority then the number of organisations on the Register involved in this activity over time could be tracked;
• Sources and types of income, albeit these are based on a sample of charities, housing associations and credit unions. Income could also be tracked over time and changes highlighted; and
• The geographical distribution of charities by local authority area. This might be useful if there was a political desire to promote voluntary activity in certain localities, for example rural areas. Again this could be tracked over time.

6.4 The main problem with the Panel is that the most detailed, and up-to-date, information held relates to voluntary organisations that have charitable status. However, the gap in information on non-charitable organisations is being addressed by the Scottish Voluntary Sector Data Management Consortium: a partnership between SCVO and a cross-section of Councils for Voluntary Service (CVS). The Consortium is developing a framework for data exchange in order to develop standard data and sampling frameworks for the whole of the Scottish voluntary sector, covering local, regional and national organisations.

Table 6.1 Networks With Information Capture Potential

6.5 The Panel was used by SCVO to send out 400 short questionnaires as part of the 2000-01 Compact Review (see Paragraphs 5.16 to 5.18). However the response rate was relatively low, at 29%. There was a particularly low response from national voluntary organisations, exactly the segment of the sector that one might expect to be interested in the national Compact. One explanation for this may be that such organisations have direct involvement with the Executive and therefore see the Compact as largely irrelevant to their needs.

6.6 The second network is made up of SCVO's members. SCVO has undertaken Surveys of its 1,600 members. On average around 400 (28%) responses are received. However membership is not representative of the sector, there being a bias towards the larger organisations and an under-representation from rural areas and the North of Scotland (SCVO, December 2002). To overcome these problems SCVO membership has been reviewed and associate membership extended to network or intermediary bodies (such as CVSs) and rural organisations in the Highlands and Islands. Given these attempts to make membership more representative, the Survey might have potential to be expanded to include questions about the Compact.

6.7 SCVO also convenes an Intermediaries Network. This brings together a range of umbrella organisations (around 25) that operate at the national level, for example the Scottish Federation of Housing Associations. The purpose is to discuss and develop a stance on matters of common interest, such as reform of charities law. The feeling is that the members are familiar with, and will have opinions on, the Compact. As such they could be used as a sounding board. However, there must be concerns as to how representative their views may be of the wider sector. There are general concerns in the literature about the domination of compacts by the "institutional voluntary sector", which includes umbrella groups (McCurry, 2001, p. 18). Were this Network to be used as the basis for some monitoring exercise then there must be well founded fears that smaller groups may feel marginalised. However, set against this, must be the fact that the Network exists and is operational.

6.8 There are a number of other databases of the sector that are being set up (Table 6.2). The main ones are:-

Table 6.2 Databases In The Process Of Being Developed

• SCVO's Workforce Panel, which is in the process of being developed and should be in place later in 2003, assuming there is co-operation from SCVO's membership. The Panel is to be a sub-set of members selected so that they are representative of the wider voluntary and community sector, in so far as this is known. The main purpose of the Panel will be to track changes in employment in the sector. It will therefore be used as a research tool, with members being given incentives to "buy-in" to it. The intention is that periodic surveys will be undertaken, using email questionnaires. The fact that respondents will have bought into the Panel is felt to be a way of obtaining a high response rate. Given that the Panel is being constructed in a way that ensures it represents the sector, it may have potential to be used as a sample frame for Compact monitoring and evaluation. However there is always a danger that those who are members of the Panel become, in effect, institutionalised so that over time membership results in their opinions diverging from those of the wider sector. This is a not major danger when this type of fora is used to collect quantitative information, which is the current intention. . However, as it is likely that much of the material to be collected about the Compact will be qualitative there could be a danger that the process of undertaking periodic surveys influences perceptions of, and reactions to, the Compact;
• Some Local Enterprise Companies (LECs) have begun to work with their local CVSs to collate databases for their areas, albeit that they do not always use the same definitions of the sector; and
• The setting up of a body to regulate and support the charity sector in Scotland, provisionally known as the Office of the Scottish Charities Regulator (OSCR). This will develop a comprehensive register of charities that are active, thereby overcoming one of the problems with other lists and databases of charities which often include bodies that are dormant. A new monitoring and reporting regime will be introduced based on the annual submission of reports and accounts.

OTHER VOLUNTARY SECTOR SAMPLING ISSUES

6.9 There are, therefore, a number of databases and information networks that could potentially be used to derive a sampling framework for monitoring and evaluating the Compact. These vary in a number of aspects, especially the extent to which they are representative of the sector. However their use also raises another question: the extent to which those included in any of these networks want to have involvement with government and therefore see the Compact as being relevant to them.

6.10 The Compact states explicit that: "not all voluntary organisations will have an interest in seeking partnership with Government. Some will prefer to pursue their own objectives without reference to the state. Others may find themselves more often in opposition to the Government than in partnership", (The Scottish Office, 1998, p. 1). Despite this statement, it is difficult to see that any part of the sector, however defined, is not affected by government at some time and in some way, especially given the broad definition of "government" in the Compact (see Paragraph 4.9). The most recent example of this is the proposed updated definition of charitable status that is to be introduced as part of the OSCR reforms. Clearly the intensity of the interface will vary and some parts, and members, of the sector will be more interested in developing a partnership than will others. However, many more may have an interest in influencing policy or legislative outcomes, regardless as to whether they see consultation or lobbying as part of a partnership development process.

6.11 It is also likely that the large national organisations within the sector already had good links to government prior to the Compact. They may therefore gain limited benefits as it is implemented. Paradoxically, it may be those parts of the sector that either claim not to be interested in the Compact, or not to have heard about it, that stand to gain the most from its implementation.

6.12 Given this it may be that the main criterion for developing a sample framework is that it is representative of the sector in Scotland, rather than being composed largely of groups that already have, or are eager to develop, partnerships and dialogue with government. Accordingly any sample should represent the sector in terms of its: size (employees and income); geographical distribution; organisational type (national, regional, local and umbrella bodies); and range of activities. Although such a sample might include groups that currently find the Compact irrelevant, the implementation process might, over time, change this situation. This is something that monitoring could track.

DEFINING GOVERNMENT

6.13 Defining parties to the Compact from government is easily done as they are listed in the Compact. However there are issues regarding:-

• The intensity of contact that the various parts of government have with the sector. At its extreme this might involve some parts of government claiming that they have no contact with, or influence over, the sector;
• Directing any survey to a member of staff who has an awareness and understanding of the Compact and can therefore give answers that are a realistic reflection of the organisation's response to the Compact's commitments. The responses to the Annual Review clearly indicated that some parts of government either had not taken on board their commitments under the Compact or the survey had been directed to the wrong person. This either reflects a lack of corporate commitment to the Compact or poor internal management structures or possibly both; and
• The definitions used by government in defining the sector. For example one of the respondees to the Executive's Mapping Exercise placed the health and social services (hospitals and day centres) within the sector. This lack of understanding, or misunderstanding, could easily bias results.

6.14 The intensity of contact with the sector could have an impact upon the response rates to any survey, with those who feel that they have limited contact being reluctant to respond. One solution to this might be to initially limit any survey to those parts of government that are felt to have the most contact with the sector. The Mapping Exercise gives some guidance on this for the Executive, something that is touched upon in Chapter 4 (Paragraph 4.11). However, the danger with this approach is that those parts of government that are not covered may feel that this absolves them from the need to implement the Compact. The initiatives that some LECs are currently making to map voluntary organisations within their areas would also seem to be indicative of a gradual move within government to embrace the sector. One can also look at the experience in Northern Ireland, where all of the Executive's 12 Departments have developed Strategies for supporting the sector as part of their commitments under "Partners for Change" (Paragraphs 3.21 to 3.26). None are able to opt out on the grounds that their contact with, and impact upon, the sector is limited. Given this, it may be that limiting any survey to a sub-set of government will give the wrong signals about the Executive's commitment to the Compact.

6.15 Difficulties in targeting the appropriate person with any survey form are by no means unique to work on the Compact. However, if it is not possible to identify a single point of entry into a particular part of government this would seem to be indicative of a lack of commitment to the Compact in that management has not responded and made implementation a priority by allocating responsibility to a named individual. If this is indeed a problem, as some of the responses to the Review would seem to indicate, then there may be merit in considering adopting the approach suggested by the Treasury (HM Treasury, 2002) in allocating responsibility to a named senior member of staff (see Paragraph 3.11).

6.16 The final management related issue is the definition of the sector. It is accepted that defining the voluntary sector accurately and concisely is difficult, even for those who are part of the sector. This seems to be an even greater problem for some parts of government (see Paragraph 6.13). Accordingly there may be a need to define the sector so that all parties to the Compact are clear about what it encompasses.

CONCLUSION

6.17 There are a variety of databases and networks from which a sample of the voluntary sector could be drawn. When making a decision the key has to be that any sample is representative of the sector. This means that it will not be biased towards national organisations and ones that already have good awareness of the Compact. The assumption is that the entire sector is affected by government and therefore it all has an interest in the Compact even if, at the moment, some may see it as irrelevant. It may be that it is this group that has the most to gain.

6.18 Government is more easily defined than the voluntary sector. However it is likely that some Departments and NDPBs are so large that they have multiple interfaces with the sector. It may be difficult to capture this through a survey and indeed it may be difficult for any one person within the organisation to be aware of all the contacts with the sector. However, if this is the case, then it needs to be seen as an internal management issue for the organisation rather than being used as an excuse to avoid participation in Compact monitoring and evaluation.

6.19 It can also be argued that, as the intensity of interaction with the sector is likely to vary, then initially only those parts of government with the most frequent involvement with the sector should be included within the monitoring and evaluation remit. The danger with such an approach is that it sends the message to those not involved that the Compact is irrelevant to them. It may also serve as an excuse for some parts of government to avoid becoming more familiar with the sector. For these reasons it is proposed that any monitoring and evaluation framework needs to cover all of the government parties to the Compact. Even those that see it as irrelevant will be forced, through engagement in the process, to become more familiar with it and hopefully more aware of the sector and its role and potential.