Introduction

This document outlines practical messages from a wide range of Learning Communities who have attempted to evaluate their plans and activities.

It aims to reflect key lessons from the range of Project Managers and practitioners, all of whom, are supporting the development of local Learning Community initiatives.

The hints and tips specifically relate to the Learning City Network (LCN) and the Department for Education and Employment (DfEE) commissioned Practice, Progress and Value Guide published in 1998. However, they have the potential to act as a valuable resource to anyone involved in evaluation.

Specific examples of activities can be found in the full range of documents in this series.

Changing Scene of Learning Partnerships

The Guide Practice, Progress and Value was produced before the advent of local Learning Partnerships and, before the proposed reorganisation of structures for Post-16 education and training in England and Wales.

In light of the development of these Partnerships and the very real effects they will have on the context in which learning communities operate, it has become clear that any revision on the Guide in its present form would be of limited use.

In these circumstances lessons from this project will compliment guidance issued to both Learning Partnerships and learning communities in the future.

In particular, LCN are working with York Consulting Ltd, who is commissioned by DfEE to undertake the national evaluation of Learning Partnerships. York Consulting Ltd drew on Practice, Progress and Value for the production of a national evaluation framework, which has been further refined to meet the specific functions of Learning Partnerships.

York Consulting Ltd's document, Priorities and Performance: A guide to self review, provides a framework for developing learning plans and evaluation issues for Learning Partnerships.

Hints for learning partnerships and those providing guidance

• There is a need for guidance to learning partnerships, as the majority of partnerships are keen to make use of all material and help available.

• There is a need to share good practice on the evaluation of partnerships.

• Material to support partnership projects would be more useful if produced as a structured work pack, including exercises that could be adapted to local circumstances.

• Two different kinds of material are needed - for strategic activity and for field centred partnerships.

• Learning communities would benefit from guidance on baseline data.

• Any guidance should make full use of examples and case studies.

For many projects, Practice, Progress and Value provided a stimulus and a reference tool.

Kirklees Education Service found the Guide a practical and useful method of evaluating progress. Despite some partnerships in the Pathfinder Projects being relatively mature, none had reached the monitoring and evaluation stage. Kirklees used the Guide to help assess priorities for action and found it was a useful yardstick. It contained practical, pragmatic strategies, which helped when partnerships floundered.

Tips from the field

What would be useful to help evaluation processes?

• Selling the benefits of undertaking evaluation to all. Evaluation is never given a high priority, even though it is regarded as important.

• Practical checklists, structured series of questions, flowcharts, prompts, matrices and sets of evaluation and self-assessment exercises, adaptable to local circumstances. These should all lead to the development of action plans with timescales and resources.

• Consultants or facilitators can offer vital support in using evaluation frameworks whilst remaining neutral.

• Evaluation guides which clearly indicate who is the target audience. Where possible provide a Glossary of Terms and details of relevant resource materials.

• Provide full details for each phase of evaluation work.

• Provide examples as to the types or sources of data that could be used as a foundation for identifying or setting targets. Where possible include examples of baseline data, possible evaluation exercises and case studies for modelling activity.

• Clearer practical guidance on how to measure the value added of partnerships. This would be useful where partnerships are particularly complex and where learning community initiatives are relatively short-lived.

Developing and sustaining Partnerships

Partnership activity was the most developed in the learning community initiatives. The following tips emerge from across the range of projects:

• Combine a shared vision with practical action and achievement to secure and retain commitment.

• Secure a "patron" or "champion", who is prepared to act as figurehead for the project to provide credibility and status.

• Have an "honest broker" at the heart of the project to prevent one particular sector from dominating.

• Involve a wide range of partners and be clear about what is required of those involved.

• Prioritise evaluation at an early stage.

• Maintain momentum and commitment. If the status of members of the partnership group decreases, then communication and high profile activities become even more important for strategic level partnerships.

• Learning community initiatives, and other partnerships, do not start from nowhere. Awareness of "history" is important. So also is the development of a sense of ownership.

• All initiatives need resources, with at the very least, a full-time co-ordinator. Recognise that maintaining the commitment of 'serious money' can be very difficult.

• Produce tangible benefits to partners and the community within a reasonable period to sustain momentum.

• Develop a set of learning communities characteristics with individuals and organisations, and review periodically to help focus priorities.

• Value the sharing of good practice and exchange of ideas with others. Use consultants to bring an outside perspective to initiatives.

• Provide some practical tips for practitioners about expectations of partnership working. For example: "don't expect instant results, have small teams who will 'do' rather than large groups who 'attend', accept failed projects as part of the learning".

Participation within the community

Vital messages emerging from learning communities engaging with community groups are:

• For the most disadvantaged communities the idea of assessing learning needs must follow considerable capacity building activity and trust building.

• In working with community groups attend to their agendas and work to protocols they set if you are serious about meeting their needs.

• Although there may be a good fit between the many kinds of community consultation and the development of participation in formal and informal learning, little has been done to make the links necessary to profit from the connection.

• Identify a separate sub-group on the role of technology at local levels.

Developing performance evaluation

Learning communities are still struggling with these issues. There is a need for clear guidance to identify the benefits of activities.

• Prioritise and find time to consider appropriate baseline data, the variables that affect different communities and how value added can be assessed in such complex initiatives.

• An absence of readily available and homogenous data makes comparative studies difficult.

• Benchmarking across single services is complex enough. The idea of benchmarking across whole communities remains a challenge.

• Evaluation frameworks must be flexible enough to respond to external change such as the impact of the emerging Learning and Skills Councils.

• Where possible, establish and share evaluation techniques at the start of a project, then build into operational plans.

• Seek advice from experts.

• Funding structures and systems still encourage individual institutions to put their priorities first. Have a clear set of priorities to help address all partners' priorities.

• Keep a collection of examples of existing practice and key questions to help with future planning.

• Consult with local people. Take time to do effective market research.

Blackburn with Darwen has developed a benchmarking framework. Socio-demographic information based upon similar population structures drawn for the 1991 Census of Population data, identified further matching criteria: levels of Indian/Pakistani/Bangladeshi residents and white residents levels of terraced housing and detached housing levels of manufacturing employees and banking & finance employees levels of residents aged 0 - 15 levels of non-car ownership and levels of two-car households levels of HND and above.

Case Study

Norwich - A Learning City

The learning city initiative in Norwich has been in existence for four years. Its aim is to create a culture of lifelong learning in the City and its travel to work area. This will contribute towards personal growth and development, economic competitiveness for individuals, businesses and the local economy and cultural and community development as a means to achieving social and economic cohesion.

The initiative was instigated by the City Council's Economic Development Unit in a climate of concern about the future of the City following a number of large-scale redundancies. Early partners were the University, local F.E. College, County Council, Adult Education Service and the Training and Enterprise Council.

Among the many successful projects that have grown out of the Learning City Initiative have been a Learning Shop, currently seeing 18,000 prospective learners a year. Other projects include a Launch Conference which engaged new partners and consulted on aims and action plans, a regular Learning Festival held during Adult Learners Week to promote learning within the City. The most recent gain has been a large thematic Lifelong Learning Single Regeneration Budget Scheme with a focus on developing a community approach to learning. The successful development of these initiatives has been one of the means by which interest has been sustained and the membership of the Learning City Group maintained.

Evaluation has been a matter of concern to the partnership since its inception. In essence two kinds of evaluation are necessary to Learning City partnerships and also perhaps to other partnership groups. The first is the evaluation of progress towards meeting the aims and objectives. The second is an evaluation of the usefulness and efficacy of the partnership itself.

In respect of the second Norwich has spent time on reviewing the partnership through a series of annual "away days". These have involved the Learning City partners devoting a half-day on an annual basis to reviewing their activities. Such evaluations have considered the continuing relevance of the Group 's aim and how this aim will be converted into action plans for the forthcoming year. The reviews have focused on the nature and membership of the partnership and what is demanded of partners. Following such discussions partners have been recruited from the media and employers and engagement sought from schools. Issues relating to the positioning of the partnership locally have also been explored. The most recent meeting spent time examining the role for the Learning City Initiative in the new system to be established by the Learning and Skills Bill.

The evaluation of progress towards meeting aims and objectives has been harder to develop. Adequate and appropriate baseline data has not been easily identified. However, in terms of Learning City focused projects, serious attempts to evaluate have characterised recent initiatives.

Obviously there are output measures within the SRB scheme and these will be used in the measurement of progress in the areas concerned.

The Learning Shop has not only counted users but has also done an in depth survey of 300 users to identify characteristics, range of opportunities sought and other relevant data. This survey is being repeated to map change, development and the engagement of disadvantaged adults.

The evaluation of the Learning Festival for 1999 proved challenging. The annual provider feedback was complemented by two new evaluation exercises. These resulted in a large-scale superficial participant profile and an in-depth study focused on a cohort of 65 participants. Although numbers attending activities were captured the quality of engagement was not captured. However, it was clear that too much was being offered over too short a timescale. Many of those participating were existing learners and there was insufficient engagement of people in hard to reach communities.

As a result the Festival in May 2000 will lead into a summer programme of targeted community based activities and events "Learning in the Community". Many of these will introduce a learning dimension to existing community Fetes and Fun-days. The idea will be to take learning information and opportunities to territory and activities where the target group feel comfortable. In this way evaluation is seen as an on-going process with indicators being identified for the year 2000 festival being identified well in advance of the event and building on the earlier work.

In essence the approach in Norwich has involved an appreciation of the value of evaluation in terms of process as well as product. It has sought to develop incrementally using evaluation in a formative way to improve and develop activity but also to move forward in terms of indicators for the evaluation process itself.