Community Development Framework

The homepage for the online resource, with quick links and descriptions for the sections and toolkits of the Framework.

This section explains the reasons local governments employ evaluation tools for community development work, as well as providing background to the Framework and Toolkit.

This section covers the fundamentals of evaluation in relation to community development work, explaining what must be considered and planned before evaluation commences.

This section provides an explanation on the program logic: what it is, how to build one and what information needs to be measured. An interactive template for constructing a program logic is included, and can be exported to a Word document.

This section provides an analysis on the many different forms and measures of evaluation. The 'gold standard' of evaluation, the randomised controlled trial, is analysed, and various interactive templates for planning an evaluation are provided.

This section details the many forms data can come in, and the various ways we can collect this data. Templates for data collection methods are provided, and can be exported as Word documents.

This section details the ways we can analyse data, how to draw meaningful conclusions from data and how we can share our findings. Templates for existing data analysis and reporting are provided, and are exportable as Word documents.

This section covers topics to consider to ensure the quality of your evaluation. A reference list is provided, along with resources for further exploration on certain topics in the Framework.

Click to open or close the Framework's dictionary/terminology search, to find definitions of the key terms within the Community Development Framework.

Dictionary/Terminology Search


Community Development Evaluation Framework and Toolkit

Section 1


This section explains the reasons local governments employ evaluation tools for community development work, as well as providing background to the Framework and Toolkit.

1.1 Why evaluate?

1.2 Project Background

1.3 Developing the Framework: Core principles

Section 2

Evaluation Foundations

This section covers the fundamentals of evaluation in relation to community development work, explaining what must be considered and planned before evaluation commences.

2.1 Defining evaluation: What is it?

2.2 Scoping evaluations: What are the key questions?

2.3 The evaluation cycle: What are the main evaluation phases?

2.4 Planning: What needs to be considered at the start?

2.5 Establishing purpose: What do we hope to find out?

Section 3

Program Logic

This section provides an explanation on the program logic: what it is, how to build one and what information needs to be measured. An interactive template for constructing a program logic is included, and can be exported to a Word document.

3.1 What is a program logic?

3.2 How to build a program logic

3.3 Identifying what information you need or "operationalising the program logic"

Toolkit: Program logic template

Section 4

Evaluation Design

This section provides an analysis on the many different forms and measures of evaluation. The 'gold standard' of evaluation, the randomised controlled trial, is analysed, and various interactive templates for planning an evaluation are provided.

4.1 Types of evaluation: Which strategies best fit context and purpose?

Toolkit: Evaluation learnings from the 'gold standard'

Toolkit: Planning an evaluation

Section 5

Evaluation Methods

This section details the many forms data can come in, and the various ways we can collect this data. Templates for data collection methods are provided, and can be exported as Word documents.

5.1 Data: Where to get it and how to assess it

5.2 Data collection: How do we measure it?

Toolkit: Methods for data collection

Section 6

Meaningful Findings

This section details the ways we can analyse data, how to draw meaningful conclusions from data and how we can share our findings. Templates for existing data analysis and reporting are provided, and are exportable as Word documents.

6.1 Data analysis: How to make sense of results

6.2 Assessment and evaluation: How to make judgements of worth or merit?

6.3 Sharing results: How are evaluation findings presented?

Toolkit: Existing data analysis

Toolkit: Reporting

Section 8


A list of all the toolkit sections in the Framework

Program logic template

Evaluation learnings from the 'gold standard'

Planning an evaluation

Methods for data collection

Existing data analysis


Additional sources and reference list

For quick access to the Community Development portal for Local Government Professionals' members, click here.

Section 1: Introduction


1.1 Why we evaluate

Local government contributes to community development in many ways, spanning across library services, place management, recreation services, cultural development, and youth services amongst others. The common aim across all areas is to improve the health and wellbeing of our communities. But how do we know we are making a positive impact? How do we know if we are making the difference we want to make? And, even if we are certain of the difference we are making, how can we systematically measure and document this?

Evaluation tools provide a way of working that helps us to understand the impact of what we do, the value of our work for others, and areas that need improvement or focus. It may identify where a change in direction is needed. It can build evidence for a particular program or service, and better communicate our work.

Some CDOs have expressed the view that while there has been a sector focus on outputs, this is beginning to shift and there is much more interest in outcomes and impact. Evaluation can help us to understand how output, outcome and impact information can fit together to improve our learning and understanding of the work we do, and how to build on that to achieve greater social impact.

An evaluative mindset

Evaluation can be messy and feel uncertain. That’s okay. Some evaluations can be neatly planned and implemented, while other evaluations are conducted in more complex conditions, and might involve experimenting, learning and adjusting as you go along.

It is important to know that there is no perfect formula for a perfectly executed evaluation, so it is best to jump in, start small, try a few things and keep reflecting and learning. Remember to keep the purpose in mind. There are many things you could measure, but focus on information that will serve you best, at a given point in time.

The goal is ultimately to approach our work with an ‘evaluative mindset’—to value evidence and learning, to ask questions, to seek more understanding of impact, to think critically and challenge assumptions, and to inform what we do and how we do it for the benefit of our communities.

"...evaluation is a highly adaptable process..."

1.2 Project Background

This project was initiated by LG Professionals WA and supported by a number of West Australian local government authorities who worked with the Centre for Social Impact, University of Western Australia (CSI UWA) to develop this resource, an associated training program and web-based resources.

What we heard

Results from a Community Development Evaluation and Measurement Survey undertaken for this project, reflect a growing desire amongst CDOs and leadership teams to increase internal evaluation capacity.

Survey responses

  • “I think there is a lot more interest in CDOs measuring impact than there was, say, 5 years ago. There is also a lot more talk around how that relates to community wellbeing which sparks that debate.”
  • “There definitely seems to be growing interest in better capturing the impact of our work…”

However, CDOs and leaders also reported resource constraints which included time, money, skills, training and capacity to support this work, and other organisational or cultural barriers to evaluation. For example, while senior leaders do give ‘in principle’ support for evaluation, this is often not translated into practice, or does not help unlock the resources required to conduct evaluations. Respondents also noted that when evaluations were conducted, they were sometimes ineffective and/or the findings were not well utilised. This tells us that CDOs are likely to need to:

Start small, utilising basic evaluation methods suitable within a resource-constrained environment.

About this Framework

This Evaluation Framework is a tailored response to the needs of CDOs—a guide to evaluation basics that can be effectively used within a range of settings.

This resource is a step towards creating a more systematic approach to evaluation across the state-wide network of CDOs, enabling CDOs and managers to develop a shared language and understanding about what is possible when evaluating their work. Outside of this resource, and through ongoing training, it is hoped that CDOs can support one another as the sector builds capacity, and that leaders can support this work too.

This resource consists of two parts:

  • Part One: An Evaluation Framework providing:
    • An overview of key things to consider when planning and managing an evaluation.
    • A foundational approach to undertaking evaluations for community development purposes in local government settings.
    • A discussion of main approaches to evaluation and methods that might be most suitable to a real-world local government context.
  • Part Two: An Evaluation Toolkit providing
    • Guidance on implementing the evaluation cycle and using different evaluation tools.
    • Templates and resources.

"...evaluation is a highly adaptable process..."

1.3 Developing the Framework: core principles

Community development work:
understanding the change we make

At the start of this project, the CSI UWA attended a Community Development Forum, where CDOs from local governments across WA were asked to summarise their work with one ‘change word’. The most popular word was ‘facilitate’, selected by one third of the 65 respondents (32.3%), followed by ‘enable’ (18.4%). Interestingly, a quick glance at all the words suggested by CDOs at the Forum reveals a strong pattern: most change created through community development work often involves a mediator, and is one step removed from the ultimate impact point.

This tells us something important about the change created by CDOs. Firstly, the impacts are created in relationship with others, and secondly, the effects of community development work are more indirect than direct. In other words, CDOs effect change indirectly, and through working with others to support them to effect change.

Figure 1: Words to capture the nature of change created by CDOs

Figure 1

Processes, social capital and outcomes

This aligns with literature on community development which proposes that CDOs often create social capital, and the conditions of empowerment and agency. 1

Phillips and Pittman (2009) note that practitioners of community development think in terms of outcomes (i.e. the “physical, social, and economic improvement in a community”), while most academics think of community development as a process or “the ability of communities to act collectively and enhancing the ability to do so”.

  • Processes of community development are patterns of action that lead to greater empowerment of local communities.
  • Social capital, or the ability to act, manifests as strengthened networks of people who feel empowered to act together more effectively to pursue their collective interests through facilitating, enabling and building capacity. 2
  • Outcomes of community development can include community groups taking action, as well as specific improvements in the community. For example, more inclusive social networks. 3

It can be useful to think of the relationship between the process and outcome as in the ‘community development chain’ illustrated below.

The process, or the way change happens, can be equally, or even more important, than the outcome. For example, programs which achieve an improvement in health indicators, but are culturally inappropriate, deliver outcome goals but would fail an evaluation of process goals. 5

Evaluating outcomes alone overlooks the importance of empowerment and participation in community development initiatives. From this perspective, it is not whether or not the outcomes are achieved that matters, it is how they are achieved that is important.

Figure 2: Community Development Chain (from Phillips and Pittman) 4

Figure 2

Implications for evaluating community development work

To understand the impact of community development programs, it is important to evaluate the state of all three components of the community development model.

At first glance this may make evaluating outcomes seem less straightforward. However, evaluation is a highly adaptable process, and what this change model means for evaluation is that there should be a focus on the causal chain of events, not just the outcomes (i.e. use of program logic). Process evaluation will be just as important as outcomes or impact evaluation, and that evaluations should be done in partnership with stakeholders—as the relationships that CDOs engage are part of the change process.

Figure 3: Community Development Chain and Economic Chain 6

Figure 3

This Framework will:

  1. Outline the use of program logic as a foundational tool for demonstration of impact.
  2. Outline the importance of process evaluation, as well as impact evaluation, for demonstrating the impact of community development work.

"...evaluation is a highly adaptable process..."

Section 2: Evaluation Foundations


2.1 Defining evaluation: What is it?

Evaluation involves collecting and analysing information to answer questions about the value or merit of an activity. Learning about evaluation will teach you how to assess the processes, activities and outcomes achieved, to determine whether programs are ‘making a difference’. 7

A formal definition of evaluation is ‘a social science activity directed at collecting, analysing, interpreting, and communicating information about the workings and effectiveness of social programs’. 7

"...ensure that all stakeholders will get some value from the information gleaned..."

2.2 Scoping Evaluations: What are the key questions?

Many systems that organisations set up around reporting are often based on rigid templates or pre-set targets and indicators. Evaluation is different and there is scope for it to be a much more open process. Evaluation is a pragmatic activity that aims to answer questions that stakeholders consider will be most useful to know, at a given point in time.

Some key evaluation questions might be:

  • Has the program been implemented as intended? If not, why not?
  • Has the program achieved its intended results? Unintended results?
  • What worked well? What didn't?
  • What worked well? What didn't?
  • What has changed as a result of this program?
  • Has this program made a difference?
  • How are the lives of (people, communities) better?
  • Does the program represent a good return on investment?

Evaluation relies on both quantitative and qualitative measurement to provide the information to answer the questions. Agreeing on a set of questions with other stakeholders normally forms the basis and starting point of an evaluation.

"...ensure that all stakeholders will get some value from the information gleaned..."

2.3 The evaluation cycle: What are the main evaluation phases?

Ideally, evaluation is built into the whole program cycle—from identifying what is needed through to designing an activity, planning, implementing, reflecting on results and determining program improvements. Importantly, good evaluation practice is to put the evaluation design in place before the program begins so that measurement occurs before, during and after the program has finished.

Figure 4 shows the interaction between cycles of evaluation and program development.

Figure 4: Evaluation in the program lifecycle

Figure 4

"...ensure that all stakeholders will get some value from the information gleaned..."

2.4 Planning: What needs to be considered at the start?

Steps in conducting an evaluation, at a basic level, are:

  1. Clarify the evaluation focus and purpose—establish evaluation questions and a program logic.
  2. Plan the process for communicating about the evaluation and engaging stakeholders.
  3. Select which tools (e.g. surveys or interviews) and measures (the way in which we operationalise outcomes) will be used for collecting data.
  4. Collect the data—before, during and after the program has ended.
  5. Analyse the data—make sense of what it is saying and draw conclusions.
  6. Answer the evaluation questions and report findings

Prior to getting started, it is good practice to create an evaluation plan.

Developing an evaluation plan

An effective evaluation plan is a dynamic tool that is updated regularly and reflects changes in the program and priorities.

An evaluation plan details the approach to an evaluation, including how evaluation results will be utilised to improve the program and make decisions going forward.

Planning an evaluation in a local government setting should include:

  1. Program logic (what is the change expected from program activities).
  2. Outcomes, indicators and measures (what difference will the program make, what can you measure to know you have made a difference).
  3. Measurement plan (when and how will you measure it).
  4. Data collection plan (with tools such as a survey or interview schedule).
  5. A list of stakeholders (including, importantly, the beneficiaries) and a plan to engage them at all stages.

It’s better to keep an evaluation plan simple and achievable, rather than complex and unachievable. Start small and scale up your approach as you build knowledge, skills and confidence.

A checklist for planning an evaluation is included in Toolkit: Planning an Evaluation, and additional resources are listed in Toolkit: Further Resources.

"...ensure that all stakeholders will get some value from the information gleaned..."

2.5 Establishing purpose: What do we hope to find out?

Establishing the purpose is key to guiding the evaluation process.

What is your purpose?

  • What information will best inform decisions and support future work?
  • Do you need to demonstrate if activities are implemented as planned?
  • Is there a need to document, communicate, and share program activities?
  • Is there an interest in exploring unintended and intended outcomes?
  • Are you intending to gauge effectiveness in localised settings before potentially expanding across other areas/wards?

Consider both the activity and the strategic context

The purpose will depend on both the nature of the activity and strategic factors.

  1. The activities that you are likely to evaluate may include community events, programs, projects, place building initiatives, social plans, or recreation services to name a few. Just as the purpose of each of these activities varies, so too will the purpose of evaluating them.
  2. What is happening in the wider strategic context that will inform the evaluation? Is there a need to focus on accountability, value for money, creating an evidence base for effectiveness?

Here are some practical examples:

  • A community garden program is popular with residents but does not have a secure funding base. The evaluation may then focus on community outcomes and whether resources are being used effectively, to convince decision-makers of its worth to communities, and to ensure it continues in future.
  • A play-based community parenting program has a strong, existing evidence base and there is a desire to implement it more widely. Due to the strong evidence already existing, the evaluation might not need to focus on outcomes as much as exploring implementation learnings before it is scaled up and rolled out in regional areas.
  • Moving support for volunteering from a physical hub location to a decentralised, online model was thought to be an innovative approach. A pilot program is trialled to gauge effectiveness. The evaluation will focus on measuring success and outcomes when compared to the physical hub model, and document processes and costs so as to compare both models and make decisions about how to progress in future.
  • A new program targeting participants from a vulnerable community focussed on employment pathways. This evaluation could employ participatory evaluation methods, allowing participants to say if they felt the program meets their needs and why. Asking questions about who did not engage and why not might also be important.

Develop your evaluation questions

It is important to ensure that your evaluation questions are linked to the purpose, that they are sharply targeted and well scoped.

Generally, evaluations have no more than five questions, although there may be sub‑questions.

Some examples of evaluation questions are outlined below: 8

  • Relevance - 'Is the activity doing the right things?'
  • Effectiveness - 'Is the activity achieving its objectives?'
  • Efficiency - 'How well are resources being used?
  • Impact - 'What difference does the activity make?'
  • Sustainability - 'Will the benefits last?'

CDOs, leadership teams and community members may be involved in scoping the evaluation questions to ensure that all stakeholders will get some value from the information gleaned, and there are no important unanswered questions at the end.

Markiewicz and Patrick suggest that good evaluation questions are:

  • Agreed: There is consensus on which questions to answer.
  • Practical: Questions are able to be answered with the resources available.
  • Useful: Inform understanding and guide future decision-making. 9

"...ensure that all stakeholders will get some value from the information gleaned..."

Section 3: Program Logic


3.1 What is a program logic?

A program logic (sometimes called a logic model) sets out visually:

  • Inputs: resources, such as staff time and venue hire.
  • Activities and outputs: actions and deliverables.
  • Outcomes: expected changes, both immediate, medium-term and long-term.

A program logic maps the expected relationships between inputs, outputs and outcomes through causal links.

For example, if people come together to create a community garden, then they will form social connections.

Links an activity to an outcome through the logic of cause and effect, and a reasonable assumption made about that cause and effect.

A program logic can help: 10

  • clarify what is to be evaluated
  • unpack the logic behind activities and expected outcomes—which helps to demonstrate outcomes
  • focus the evaluation questions
  • identify potential sources of information and inform data collection methods
  • assist with interpreting findings and drawing conclusions

A program logic helps you think through outcomes, medium-term outcomes and flow-on impacts. For example, if ‘reduced social isolation’ is a stated goal, short-term outcomes might include repeated attendance at an organised meeting place, medium-term outcomes might include new connections being facilitated and long‑term outcomes might include sustained, regular and meaningful connection and engagement with local community members after year one.

Figure 5 helps to explain outcomes via a simple graphical representation of the different elements of a program and how they are linked together.

Figure 5: Elements of a program logic

Figure 5

"A program logic maps the expected relationships between inputs, outputs and outcomes through causal links..."

3.2 How to build a program logic

A program logic can demonstrate, through causal links, that particular outcomes are likely to be achieved even if you are unable to measure all outcomes or long-term outcomes (which can be difficult to measure).

Program logic will always include:

  • Inputs (resources)
  • Activities
  • Outputs
  • Outcomes

Developing a program logic will also help to identify what to measure and when. This can help to clarify what information you need to provide evidence for.

A program logic could also include:

  • Participation
  • Assumptions
  • External factors which could impact the program

Program logic is best developed in a team that includes partners, stakeholders and leaders. This will ensure it captures different perspectives and ideas. Ideally, a ‘co-design’ approach can be used which will also include beneficiaries and community representative/s. This approach can provide lived experience insights which may help clarify or correct any assumptions.

A review of evidence to validate the causal links and assumptions in your program logic can significantly strengthen your logic model.

This evidence review can consider different fields of knowledge, or different lenses. For example, you can validate links using what you know from documented lived experience perspectives, a scientific lens (using for example systematic reviews of the evidence) and/or a policy lens. In many cases these multiple lenses will reinforce one another.

Considerations for developing a program logic:

  • Engage a broad stakeholder group.
  • Use co-design methodology to draw out lived experience.
  • Draw upon existing evidence, such as scientific knowledge from systematic reviews.

Ultimately, use logic based on cause and effect: if this happens, then that will be a consequence.

e.g. Building program logic

A school-based breakfast program aims to improve educational outcomes for at-risk students. While it may seem a stretch to suggest a simple breakfast program with minimal funding can achieve educational outcomes, these causal links can be mapped using a program logic and validated through existing knowledge.

Building program logic diagram

In the above example, you could consult with participants (lived experience) to validate whether it is true that the chance to have breakfast at school does motivate them to attend. Then, you could validate the link between increased attendance and improved literacy and numeracy outcomes through a literature search. Links between attendance and improved educational outcomes are also likely to be validated and politically endorsed in the policy literature and key frameworks available in the education sector.

This maps a pathway between cause and effect, to help determine whether progress is as expected and, if not, what might be affecting it. It is beneficial to return to your program logic during the implementation process. It is a useful tool to reflect with others upon any changes that have occurred.

Closely related to program logic is a theory of change. While a program logic visually sets out the different elements of a program and establishes their links, a theory of change is a statement which explains the rationale or reason for the linkages between elements of the program logic and the expected change. A logic model describes, while a theory of change explains. With both program logics and theories of change there may be great variation in the extent to which underpinning evidence already exists. In some cases the theory or logic may as yet be untested. In other cases the theory is well established, and the theory of change will make this explicit.

"A program logic maps the expected relationships between inputs, outputs and outcomes through causal links..."

3.3 Identifying what information you need or "operationalising the program logic"

Once a program logic has been developed, identifying what to measure will be easier.

Performance information should be sought across all parts of the program logic including inputs, outputs, short-term or immediate outcomes, and if possible, longer‑term outcomes and impacts too. This is sometimes referred to as ‘operationalising’ the program logic.

Operationalising outcomes involves considering an intended outcome and asking, how do we achieve that? How do we measure that?

Consider your intended outcome and identify what instruments or measures might be used (e.g. for ‘wellbeing’ you might consider the World Health Organisation Quality of Life Instrument).

In this process, the theoretical concepts within your program logic (e.g. eating a good breakfast will result in improved engagement with school) are turned into specific variables that you can measure (e.g. increased attendance rate) or for which you can even set a target (e.g. a 50% increase in school attendance or a 15% improvement in school performance). Data may be from existing data sources or data collection that occurs as part of your evaluation, or a mix of both.

Identifying data sources

A practical way to begin is by identifying what information is already being collected. Input and output data will be easy to access and collect as part of the management and administration of the activity. For example, project officers should be collecting records of room hire, number of participants who attended the workshop, number of workshops held, as part and parcel of their management of the program.

Outcome data may require more dedicated data collection efforts. Within your program logic you may have developed outcome statements such as: “students are more engaged in school as a result of attending the program”. A good indicator for this might be attendance rates which are already collected by schools. This is a very reliable data source and should be easily accessible.

However, for many immediate and longer-term outcomes there is often a need for planned data collection.

Outcomes measurement asks questions about change in a way that attempts to link the changes being measured to the program itself. You may ask participants directly, for example, what they have gained from the workshops. Questions asked might include: What has changed in the lives of individuals, families, organisations, or the community as a result of this program? Has this program made a difference? 11

See Section 4: Evaluation design for more information on data sources and measurement.

e.g. Operationalising the program logic

Operationalising the program logic takes outcome statements (such as increased engagement in community centre programs) and translates them into context-specific indicators (such as a 50% increase in enrolments) and measures (e.g. registration forms). A worked example of a parent education program (up to the point of determining appropriate measures) is below.

Figure 6: An operationalised program logic for a parent education program 12

Figure 6

"A program logic maps the expected relationships between inputs, outputs and outcomes through causal links..."

Program Logic Template

Developing a clear program logic establishes a connection between the resources and inputs related to an activity, and the outcomes and impact it intends to achieve. When designing an evaluation, the program logic will help inform the data collection methods and key evaluation questions. It also provides an opportunity to identify external influences upon a program’s outcomes, as well as potential unintended consequences (both positive and negative).

There are many ways to set out a program logic, however the layout matters far less than the ability to clearly represent the cause-and-effect relationship between inputs, activities and outcomes, and whether the diagram makes sense to its intended audience.

Below is an interactive template for creating a program logic with the following functionality:

  • Add rows to the table by clicking the "Add new row" button.
  • Delete the last row by clicking the "Delete last row" button. This will permanently remove all text in the deleted row.
  • Edit the title and add text to the table cells by clicking on them.
  • Export the table as a PDF document or Word document using the relevant buttons
  • Text entered into the table will remain if you switch sections, but not if the website is reloaded.
  • Scroll down to find a worked example of a program logic.

Program Logic Title

Inputs Outputs Outcomes Measures
Activities Participation Short-term Medium-term Long-term

External factors that may influence outcomes achieved:


Worked program logic example: 'Together 4 Tea'

Program Logic: 'Together 4 Tea'

Inputs Outputs Outcomes Measures
Activities Participation Short-term Medium-term Long-term
Suitably skilled staff to facilitate initial meeting Initial meeting of 'Together 4 Tea' groups and facilitator Community members aged between 65-80 who are experiencing social isolation Have regular social engagements planned Form connections within local peer group Significant increase to social capital Doctor's visits reduce: within nine months of the first 'Together 4 Tea' meeting, a 5% reduction in visits to the GP.
A range of appropriate venues Follow-up meetings of 'Together 4 Tea' groups Local café owners who opt in to provide a venue Increased number of casual connections within the local community Slightly improved health and feelings of wellbeing Significant improvement to health and wellbeing Fewer hospitalisations: within nine months of the first 'Together 4 Tea' meeting, a 10% reduction in hospital admissions amongst participants.
Funding Organising group meetings, including rotating participants who opt for a new group Commitment to attending regular meetings Friendships are formed and sustained Social capital: within nine months of the first 'Together 4 Tea' meeting, participants report a significant increase in their network of friends.
Participants plan their own, ongoing and frequent meetings and activities Organising engagement: continuing engagement with at least one other participant after six months

External factors that may influence outcomes achieved:

Pre-existing medical conditions that may prevent regular attendance at 'Together 4 Tea' meetings. Lack of available transport which might prevent attendance at meetings.
Previous disputes within sections of the community which might impact the ability for group members to form connections. Prevailing public health concerns that prevent group meetings.

"A program logic maps the expected relationships between inputs, outputs and outcomes through causal links..."

Section 4: Evaluation Design


4.1 Types of evaluation: Which strategies best fit context and purpose?

There are many types of evaluation which cater to different purposes. For community development work, generally three types of evaluation will be particularly relevant:

  • Process evaluation: describe the implementation.
  • Outcome or impact evaluation: measure the changes that resulted from the intervention.
  • Economic evaluation: how resources are allocated, or benefits calculated from costs.

Depending on what is critical to find out, a process evaluation might be emphasised more than an outcomes evaluation—or vice versa. However, generally a little of both would be recommended, as shown in Figure 7 below.

Figure 7: Process and outcome evaluation

Figure 7

Process evaluation

Process evaluation can be ongoing, periodic or final, and it asks how—or how well—a program has been implemented. Process evaluations may be used to measure success, inform changes and improvements to the program or activity (action learning model), as well as helping to guide future planning.

Outcome evaluation

An outcome or impact evaluation assesses the extent to which a program achieved its objectives (outcomes) through the intervention (or activity). It may go further than looking at outcomes and try to examine attribution—for example, to what extent have the changes seen resulted from the intervention?

The term outcome evaluation is sometimes used interchangeably with ‘impact evaluation’. Impacts, however, are typically considered as longer-term outcomes and often imply a broader reach than the immediate activity—such as community impacts.

A key concept in outcome/impact evaluation is the ‘counterfactual’. The counterfactual simply refers to what things would have looked like if the activity or program did not take place. When looking for evidence of outcomes or impact it is important to compare outcomes resulting from what actually happened with what would have happened if the program or activity had not taken place.

This comparison approach is at the heart of rigorous research, typically found in the realm of medicine and science.

Economic evaluation

Economic evaluation can help decision-makers assess how to best allocate scarce resources (CDO time and skills, equipment, premises, energy, etc.).

Economic evaluations measure both the costs/investment of a program and the outcomes of the program. Resources alone will not tell us if a program is cost‑effective, so an economic evaluation must consider the costs alongside the outcomes of a program.

Some types of economic evaluations are:

  • Cost-effective analysis - which compares the investment made in the program with the outcomes achieved.
  • Cost-benefit analysis - which uses a systematic approach to estimate the costs and benefits in dollar terms associated with outcomes so as to determine the benefits relative to cost.
  • Social return on investment (SROI) - which is an applied practical form of cost-benefit analysis using proxies of social, environmental, economic and other values and strong stakeholder engagement.

Local governments may use economic evaluations to decide how to invest when presented with programs with very different aims that may be hard to compare otherwise. For example, a cost-benefit analysis may help inform whether it makes better economic sense to invest in sporting or recreation infrastructure programs for enhanced community health and wellbeing.

Economic evaluations inform only one element of decision-making. They do not necessarily tell us which option is “better” in terms of delivering effective outcomes that are most meaningful to communities. If the cheapest option is also the most effective, then it is clearly the most cost-effective> option. However, in practice, options may offer different compromises. For instance, if we pay this much more, we could get an additional ‘x amount’ of effectiveness. This is also known as an incremental cost-effectiveness ratio (ICER)—the additional cost incurred per additional unit of effect accrued.

Programs might be valued for the effects or benefits that cannot be expressed in economic terms, which can often be the case with community development interventions. Economic evaluations are just one part, albeit an important part, of the bigger decision-making picture.

Other types of evaluation

While process and outcome evaluations are the bread and butter of evaluation approaches, there are many other types of evaluation:

  • Developmental evaluation can provide useful tools for situations that are in a state of development, flux or uncertainty. In such situations program logic might be difficult to establish because there are so many moving parts or the theory/ assumptions underpinning the model are also shifting. Developmental evaluation is purposefully adaptive and contextual, and supports innovation. It involves asking questions throughout the process and feeding the findings into a program in real time or close to real time. This evaluation approach has similarities with participatory action research approaches.
  • Participatory action research (PAR) seeks to situate power within the research process with those who are most affected by a program and involves the active involvement of all stakeholders including program clients, practitioners, and community members.
  • Social impact assessment (SIA) considers the various positive and negative impacts on a community that may arise from a particular initiative or project. It provides a process for identifying, analysing, assessing, managing and monitoring potential social impacts over the stages of the project lifecycle.
  • Realist evaluation is a philosophical approach to evaluation that calls on evaluators to ask not just what works but what works for whom and under what circumstances. Both program structure and context matter.

What type of evaluation to use?

It is not necessary to understand every type of evaluation, but rather the scope of what tools are available. Evaluation can be responsive to different contexts and needs. There is no single right way to do an evaluation.

As you increase your confidence and skill, you may find yourself:

  • making more decisions about the emphasis placed on levels of participation and empowerment.
  • questioning assumptions rather than relying on the existing evidence-base.
  • using more robust instruments or complexity-sensitive instruments.
  • using responsive rather than rigid measures.

Understanding process and outcomes evaluation types provides a solid foundation, while recognising the need to be flexible and make any changes that seem necessary and justifiable.

"There is no single right way to do an evaluation..."

Evaluation Learnings from the 'Gold Standard'

This section of the Toolkit provides an overview of the ‘gold standard’ of measurement—the randomised controlled trial—and an associated method, the quasi-experimental approach.

Community development work does not happen in a laboratory where factors can be isolated and controlled easily. The complexity of social and community interventions, the indirect effects and nature of the work, as well as resource constraints, will mean that there may not be opportunities to conduct meaningful quasi-experimental designs. However, this method provides important foundational understanding for research and evaluation practitioners.

Randomised Controlled Trial

To gather evidence for the effectiveness of an intervention, there is no better or more robust approach than the randomised controlled trial (RCT). An RCT measures the effect of an activity by measuring outcomes achieved, and then comparing that to a measure of what would have happened if the intervention had not been applied (i.e. the counterfactual).

In an RCT, two groups of randomly selected participants are separated into a control group and an experimental group. The experimental group receives the intervention while the control group does not.

The random selection process is an effective way to remove bias, especially in high sample sizes, to ensure the two groups start out with relatively the same composition or mix of characteristics. This means that outcomes in the experimental group can be more easily attributed to the intervention. If the sample has been randomly selected, it is possible to generalise the results to other populations (providing the sample is large enough).

This design represents the ‘gold standard’ approach—the approach demanded, for example, from peer-reviewed journals or medical research, where stakes are high and evidence needs to be especially robust.

Figure 9: Example randomised controlled trial framework (n = sample size)

Figure 9

Quasi-experimental Design

As social interventions are more complex and multidimensional than medical interventions, social research often involves an adapted form of the RCT, known as a quasi-experimental approach. A quasi-experimental approach acknowledges that a pure RCT is not possible in social contexts, but will still draw upon a similar design (through the use of a control group or control data) to find evidence of impact.

To successfully implement a quasi-experimental design, a specific set of conditions is required which include:

  • program design and evaluation design occur simultaneously (at the start)
  • the ability to capture baseline data (or pre/post data) to measure change over time
  • access to control data, or a control group
  • resources to engage and collect data from a control group
  • adequate sample size of intervention participants and control group participants
  • enough time for change to be realised, and for evaluation methods to capture this change.

Measuring differential impact: Control groups and control data in social settings

The problem to resolve when researching in social settings is how to create a control group of randomly selected participants? It would be unethical to recruit participants for a program and then deliberately deny them the chance to participate, especially if that program meets an important need.

In community programs participants mostly self-select into programs, which means the participants might have different characteristics to non-participants in ways that will affect outcomes.

e.g. Randomly selected participants

For example, people who register for ‘Together 4 Tea’ may be naturally more interested in socialising and being socially connected compared to people who do not sign up. So even before outcomes are measured, the intervention group may be more likely than the control group to be responsive to social interventions and to self-report feelings of social connectedness.

One approach to resolving this is the ‘difference in differences’ method (also known as the DID or DD method). In this method, we compare how much each group changes over a period of time with respect to a certain outcome and then compare the extent of the change between the two groups.

Where it is not possible to access a control group, or where there are no resources to measure outcomes across a control group, it may still be possible to use control data—for instance, by using published population-level statistics relating to the outcome you are measuring.

In this way, it is possible to determine how the outcomes achieved by the initiative compare to the counterfactual (the outcomes achieved by doing nothing).

See Toolkit: Data collection for sources of data and measures.

e.g. An example of the difference in differences method

Let’s say you are running a program to reduce rates of smoking in your community. How can you measure that rates did decline and, in particular, rates of smoking for people participating in your program (to demonstrate program effectiveness)? It will be easy enough to collect data from program participants that may show smoking rates decreasing. But without a control group or control data we do not know if this outcome was because of our intervention or due to other factors.

Societies are complex, and many factors impact upon outcomes. For example, in Australia, smoking cigarettes sharply decreased over a period of time, as a result of many different factors including public health campaigns targeting children to prevent smoking, GPs encouraging people to quit smoking, rising costs of smoking, and an increased interest in healthy lifestyles.

The following graph uses fictional data and a fictional ‘quit smoking’ program to demonstrate the ‘difference in differences’ method that can be applied in such a case.

Figure 10: Rates of smoking over time in program participants versus total population (illustration only)

Figure 10

Using this approach, we can see that those who received the intervention appeared to have reduced smoking at a greater rate than those in the total population—giving strength to the claim that the intervention was effective. Thus, the difference in differences method can demonstrate the measurable effectiveness of the program.


Whilst these evaluation designs will provide the most robust evidence, in reality, evaluators must be pragmatic. Effectively measuring change over time in an intervention group, as compared to a control group (thus measuring the differential impact), is very rarely implemented fully in a social context.

As one Community Development Measurement and Evaluation survey respondent commented, “Evaluation is identified as important by most of the team but it’s impossible to incorporate and complete a comprehensive, relevant, meaningful, evaluation process with inadequate resourcing and processes which then allow for linking evaluation recommendations to planning future work with the community.

A realistic approach can involve a mix of methods which provide enough quality information to inform decisions, even if the evidence is not strong enough to be generalisable or to be published in a peer-reviewed journal.

"A realistic approach can involve a mix of methods which provide enough quality information to inform decision…"

Planning an Evaluation

Steps in evaluation planning

There are many ways to approach evaluation planning. One example, adapted from the NSW Government Evaluation Toolkit, is set out below, and includes seven key steps.

  1. Create a program logic.
  2. Develop the evaluation plan.
  3. Assemble the evaluation team or commission a consultant.
  4. Develop the evaluation design.
  5. Develop the evaluation work plan.
  6. Manage the implementation of the work plan.
  7. Disseminate evaluation findings.

What to include in an evaluation plan

The following provides a list of what things you should consider including in an evaluation plan.

See also Section 2: Evaluation Foundations for further information.

About the program/activity

  • Title
  • Alignment to Strategic Community Plan/Corporate Business Plan
  • Location/s
  • Team Members
  • Lead department/team
  • Key stakeholders and beneficiaries
  • Objectives
  • Program logic
  • Budget (including evaluation)

Framing the evaluation

  • Purpose and focus of the evaluation: the 'why'
  • Evaluation questions: what in the strategic context is important, what questions are you trying to answer?
  • Evaluating approach: process or outcomes focus
  • Monitoring requirements
  • Considerations: what particular considerations led to the chosen approach?

Resource requirements

  • Personnel
  • Budgets (also consider budget to reimburse participants involved in lengthy interviews or focus groups for their time in this section)
  • Data storage and collection systems

Potential risks and ethical issues and how they will be managed

  • Who are the vulnerable groups or individuals who might be impacted by the work?
  • What unintended consequences of the work could arise?
  • What might go wrong at different stages of the work?
  • What might prevent the project from being delivered on time and within budget?
  • What negative impacts could the work have upon different groups?

Analysis and synthesis

  • What standards will be used to assess merit (are there targets or benchmarks already set, will you make judgements based on sector standards, or criteria co-designed with lived experience advisors)?
  • Who will be involved in analysing data and reaching conclusions?
  • How and when will analysis happen?

Templates for planning

Using table-based templates can provide a quick way to capture the key information required within an evaluation plan and can also be used to set up reporting and work plans. The following are template resources for an evaluation plan, reporting plan, and work plan with example entries based on ‘Together 4 Tea’

Below are interactive templates with the following functionality:

  • Add rows to the table by clicking the "Add new row" button.
  • Delete the last row by clicking the "Delete last row" button. This will permanently remove all text in the deleted row.
  • Quickly remove the example cells of a table by using the respective "Delete last row" buttons.
  • Edit the title, headers and cells by clicking on them.
  • Select the tables you want to edit and hide the ones you don't using the toolbar.
  • Export the visible tables to a Word document using the relevant button.
  • Text entered into the table will remain if you switch sections, but not if the website is reloaded.

Evaluation plan template

Evaluation questions - typically no more than 5 questions Focus of evaluation How will data be collected and when? How will data be stored and who will have access? Who is responsible
E.g. How effective is 'Together 4 Tea' at reducing social isolation in 65-80 year olds? Identifying any reduction of prevalence and/or severity of social isolation amongst the target group. Focus groups and surveys will be conducted to gather baseline data prior to first meetings.
Data will be collected in the same manner after the first round of four meetings, and again after nine months
Data will be stored on the local government's secure drive and will be accessible only to the evaluators. The community development manager will be responsible for oversight of the evaluation plan, however the community development project officer will be responsible for designing and implementing the evaluation.

Reporting plan template

Type of report Audience Content Frequency/due dates Who writes the report Who approves final report
E.g. Interim evaluation report. Local government senior managers, project sponsor and funders. Initial findings and recommendations for program iterations. Interim evaluation report to be delivered three weeks after the end of the first round of 'Together 4 Tea' Community Development Project Officer Community Development Manager

Work plan template

Yr Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
1 E.g. Survey and focus group design Baseline data collection. Follow-up data collection and data analysis (survey and focus groups). Interim report developed. Follow-up data collection (survey and focus groups) and data analysis. Final evaluation report published.

Process evaluation measurement plan

Evaluation questions (Developed from program logic) Focus of evaluation (What do we want to know?) Indicators and measures (How will we know it?) Data collection (How will data be collected?) Frequency (When will data be collected?) Responsibility (Who will collect the data?)
E.g. Has the implementation supported the development of new social connections for participants? What has 'Together 4 Tea' done to aid the development of new social connections? Participants are forming new friendships.
Participants are socialising with one another outside of the program.
Data will be collected through surveys, delivered by post, with pre-paid, self-addressed return envelopes. Data will be collected after the first round of 'Together 4 Tea', and at the end of each round when changes have been made to the program. Community development project officer.
(Initial meeting facilitators should not facilitate focus groups, to reduce risk of confirmation bias.)

Outcome evaluation plan: What to measure?

Evaluation questions (Developed from program logic) Focus of evaluation (What do we want to know?) Indicators and measures (How will we know it?) Data collection (How will data be collected?) Frequency (When will data be collected?) Responsibility (Who will collect the data?)
E.g. Have 'Together 4 Tea' participants experienced a reduction in the severity of social isolation? What are participants' experiences of social isolation like now, in comparison to prior to the program? Self-reporting decreased experience of social isolation.
Improved health indicators over time.
Baseline data collected via postal survey.
Follow-up data collected by postal surveys and focus groups.
Baseline data will be collected prior to the initial meeting.
Follow-up data will be collected after the first round and again nine months after the first round.
Community development project officer.
[Note: Initial meeting facilitators are not to conduct focus groups due to risk of confirmation bias.]

"A realistic approach can involve a mix of methods which provide enough quality information to inform decision…"

Section 5: Evaluation Methods


5.1 Data: Where to get it and how to assess it

Evaluation involves using performance information or data to answer the evaluation questions you have developed. As detailed in Section 3.3, a program logic can help us to identify the performance information we need. This section takes a closer look at data sources and collection methods.

Before deciding on the data collection methods to use, it is important to consider:

  • What are the key evaluation questions to be answered?
  • What information sources and/or measurement instruments already exist?
  • What are the most appropriate and practicable methods for collecting new data?
  • What resources do different methods require and are there adequate resources available to collect and analyse the data?
  • What risks and ethical issues need to be considered?

It is also worth thinking about how you will define and understand different data. For example, it seems straightforward to count participants, but what do we mean by a ‘participant’? Is this a person who registers their interest in a program but does not attend, someone who registers and pays but only attends once, or someone who registers and attends at least 60% of the time? Any definition is possible but it is worth creating a meaningful definition that all stakeholders can work with and that you can use consistently throughout your evaluation.

This helps to provide consistency in the collection and use of data, make data easier to analyse and preserve knowledge of what you did so it can be replicated or used later on. A formal way of doing this is through a data dictionary which describes the meanings and purposes of data elements within the context of a project.

Quantitative and qualitative data

Data can be quantitative or qualitative.

Quantitative data> refers to data that is measured in terms of numbers and counts and can be sourced from administrative sources and survey questions.

Qualitative data is more narrative or text-based and collected through methods such as case studies, observation, focus groups, and stories of change.

A mixed methodology is often recommended and includes both quantitative and qualitative data and methods of analysis. For example, surveys often include quantitative measures (e.g. scales of agreement or satisfaction, measures of income, and wellbeing) as well as qualitative measures (e.g. open-ended questions about what participants most liked about an activity).

Data sources and collection

Different types of data sources and analysis include:

  • Academic or published knowledge - published in research journals or collected through systematic reviews. For instance, systematic reviews from the Campbell Systematic Reviews and the Cochrane Library.
  • Lived experience - the lived experience of community members, specific cohorts or beneficiaries.
  • Administrative data - collected within the local government, community services you partner with, or as part of the program you are running.
  • Evaluation data collected - collected as part of the measurement and evaluation methods. Depending on what you are measuring you may develop your own methods or you may want to identify if there are existing available measurement instruments for the outcome you want to measure—for example, the World Health Organization’s Quality of Life Scale.

Data collection methods include:

  • surveys
  • interviews
  • focus groups
  • case studies
  • analysis of existing data
  • document reviews

Evaluation data can also be drawn from:

  • program documentation
  • visual records
  • social media
  • storytelling
  • observation
  • community meetings

Any information that is collected systematically (and, ideally, consistently too) that you can make sense of in relation to the evaluation questions can be considered as data. Often there are limitations (such as biases) with the data that an evaluator must rely on. As an evaluation practitioner it is important that you are aware about such limitations. Having a rich variety of sources can help to counter the limitations of any one particular source.

For information on different data collection methods:

Evaluation Guide, 2015, Government of Western Australia

BetterEvaluation website

Administrative data

Administrative data is information collected by government, business and other organisations for purposes such as record keeping, registrations and completing transactions.

Administrative data can be used in a number of ways. Data collected around certain populations can be extracted for comparison with a participant group.

Examples of administrative data include hospital admissions, school attendance, Medicare enrolments and police attendance.

Administrative data collected over a period of time can provide a basis for longitudinal evaluation. Caution should always be taken due to the possibility of missing data, data quality, access and data protection.

Be careful to understand how terms are defined as well—so you know what the measures mean!

Data quality

The ability of data to answer evaluation questions depends on the quality of the data.

Data quality is often assessed in terms of the following criteria:

  • Relevance: the data must add meaning and meet the needs of the evaluation.
  • Accuracy: the data must have fidelity to what it is supposed to represent, and be truthful.
  • Timeliness: the data must be up-to-date and available.
  • Coherence: the data must be comparable, reliable and consistent over time.
  • Interpretability: the data must be able to be understood and utilised
  • Accessibility: the data must be easily accessible for those doing the evaluation
  • Validity: the data must precisely reflect what is intended.

Triangulation is when more than one method is used to collect data on the same question. It is a way to reinforce or examine results to assure rigorous and valid findings.

e.g. Triangulation:

For example, an evaluation of a community garden initiative may include a survey of all participants as well as in-depth interviews with a selection of participants.

Data from both methods is then compared to see if different sources of data confirm the same findings.

Validity and reliability of data

Validity refers to the extent an instrument or measure accurately measures what it intended to measure. We can consider both face validity (does the measure appear to adequately measure what it is intended to) and content validity (when thinking of the agreed definition of a particular construct/idea such as ‘wellbeing’ would experts and others agree that the measure captures the meaning of the construct).

Reliability> refers to the extent to which an instrument produces consistent results.

There are three types of reliability common in program evaluations:

  • Test-retest reliability: does an instrument produce similar results with repeated testing?
  • Inter-rate reliability: do two or more people administering the instrument produce similar results?
  • Internal consistency reliability: do survey items that are intended to measure the same characteristic correlate?

External factors impacting outcomes

It is important to think about external factors that impact on the outcomes observed in an evaluation, and collect some information about this as you go (even if the information you collect is only notes). For example, when evaluating the effectiveness of a breakfast club program to improve school attendance, it would be important to know what other interventions are also targeting school attendance, as well as any other factors which might have a bearing upon the results. An example of other interventions might include an incentive program being run by a local sporting club to encourage students to attend school.

It is important to consider external factors that might influence outcomes when collecting data, so you can fairly assess the effectiveness of an activity.

At times your context may be quite complex with many factors influencing the outcomes all at once, within a general environment of change.

Keeping notes about these changes as your program unfolds is called situational analysis.

e.g. The 'Your Move' program:

During COVID-19, many factors in the external environment may have influenced your program, so it is a matter of documenting all the factors that may have contributed to the outcomes.

For example, the ‘Your Move’ program, which involves the City of Stirling, looks at the outcome of encouraging people to use the train. However, during the COVID-19 period there were many changing external conditions that would have influenced outcomes, for example:

  • fears of infection from public transport
  • lockdown and closure of businesses in the CBD
  • more people working remotely
  • introduction of free parking in the city

"Data collection and analysis can take a long time, and this should be taken into account when planning interviews..."

5.2 Data collection: How do we measure it?

Once you have reviewed your program logic (and evaluation questions), you will have a better idea of what existing data you want to access and use. You may also decide that you need to collect further information as part of the evaluation.

Analysis of existing data and evidence

Data and evidence can be obtained from sources such as project documents, government records and publicly available statistics. The Campbell Systematic Reviews (Campbell Collaboration) and Cochrane Library provide access to systemic reviews, which can be useful sources of knowledge.

Common data collection methods

A summary of some methods to choose from is provided here.

Remember: When collecting data, you need to consider informed consent. At a minimum, people should be told why the information is being collected, how it will be used, and how confidentiality will be protected.

Document reviews

Data and evidence can be collected from existing documents for the purpose of background information.

Case studies

A case study is a detailed investigation of a person, group or event in its real-world context, over a period of time.


Almost all written surveys are now conducted via web-based platforms such as Survey Monkey, which you can access for free.

Surveys are convenient and can be anonymous.

When surveys are used to gather information from a large number of people, they can provide statistical findings that may be generalised to the broader population.

When undertaken at various intervals (e.g. prior to, during and after program implementation) the data can capture change over time.

Surveys give limited opportunity to gather in-depth information, and low response rates and small sample sizes will limit your ability to generalise findings, or do meaningful quantitative analysis. (As a very general guide, generally aim for a sample size of 30 people or more to make a survey worth doing.)


Interviews can be conducted in person, by telephone or online. Consider which setting will best ensure respondents are comfortable sharing information. The mutual sharing that happens in an interview encourages the building of trust and can facilitate information gathering from sources who may otherwise be difficult to access.

In-depth discussions can uncover deep insights and unique perspectives and, as such, interviews can help understand why and how an intervention has worked. Interviews are suitable for use at any stage prior to, during, or after a program.

While in-depth views and opinions are easily accessed through this method, gathering large quantities of data through interviews can be resource intensive. Data collection and analysis can take a long time, and this should be taken into account when planning interviews.

Focus groups

Focus groups are an informal way of interviewing groups of typically 6–12 people, and usually last for around 90 minutes. They allow people to express their opinions and ideas freely and encourage open expression due to dynamic conversation created through multiple perspectives. Group discussion is rich and can complement or build on quantitative survey. Focus group members can be presented with results to be interpreted and emerging patterns to be explored.

Focus groups can help build understanding of why and how an intervention has worked or might work, and are suitable for use at any stage prior to, during, or after a program.

Given the short timeframe and nature of focus group discussions, there is little opportunity to cover many topics in this setting. A broader range of ideas can be discussed in a one-on-one interview setting.

Selecting a method

A number of factors will impact upon the types of methods used to conduct an evaluation, including:

  • The stage of the program being evaluated:
    • Is the program already underway? This gives you scope to access participants with some degree of ease.
    • Is it already complete? Consider how you will access participants and how many might respond to a request for data collection.
    • Is it about to enter the planning and design phase? This gives you the greatest scope to design an evaluation that will provide the strongest evidence on impact and implementation. Designing your evaluation prior to the start of the program is the preferred approach as it gives you the best chance to establish pre-post data and develop comparison reference points.
  • Timeframe:
    • Is there scope to capture data at multiple points in time?
    • How quickly does the evaluation need to be developed in order to access participants during program delivery? Methods such as interviews can be more time consuming, while surveys can be turned around faster.
    • If the program operates in cycles, data collection can be designed around this, and feed into iterations of the program.
  • Stakeholders:
    • Are there accessibility considerations which might impact respondents’ ability to participate? Certain formats may be more appropriate in certain instances. For example, interviews might be more culturally sensitive in Indigenous communities when approached in an appropriate manner. Include a broad set of stakeholders in your evaluation planning and always include a role for beneficiaries of programs.
  • The type of data sought:
    • A mixed methods approach is ideal as both quantitative and qualitative data will provide more robust evaluation findings.
  • Prior evaluation experience:
    • It is okay to select methods based on what you feel most comfortable using. A ‘good enough’ approach is accepted by expert evaluators: real world constraints will always be present. Start simply and build capacity with experience, while knowing that lack of prior evaluation experience is not a barrier to conducting an evaluation.

"Data collection and analysis can take a long time, and this should be taken into account when planning interviews..."

How to find and assess existing data

Assessing existing measures - questions to ask:

  • How reliable and valid is the data source?
  • Is it fit for purpose?
  • Do you need to translate it to Australian or other context?
  • Is the measure well-suited to the target population?
  • Is it culturally appropriate and relevant?
  • Are staff members qualified and able to administer?

Data and Research Sources:

Australian data and research sources:

World data and research sources:

Existing measures or indicators of potential use - where to look:

Methods for collecting original data


How to Use Pros Cons

Surveying is sometimes used over the course of a project/evaluation to provide information on a large number of participants, often over a period of time.

Depending on the method used, they can be an effective and low-cost way to collect data.

There are five key methods for surveying: face-to-face, postal, online, telephone, or handout. The efficacy of each varies depending on intended participants.

Typically provides detail, giving information on a variety of topics.


Easy to collect statistical data over periods of time.

Risk of sample bias: certain types of people are less willing or able to participate, leading to a gap in the data.

Measurement error: data collected may be incorrect where participants provide false information (misinterpret the question, imperfect recall, etc).

Risk of bias in question: leading questions can result in higher likelihood of certain results.

Click below to view an example survey template that can be exported as a Word document.

Example survey template

Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree
It was easy to find information about this program
The program ran at times that made it easy for me to get involved
People running the program made me feel welcome
Information was always given to me in a way that made sense
The program was interesting and useful to me

Interviews and Focus Groups

How to Use Pros Cons

Often used in process evaluation. Interviews and focus groups are useful for collecting qualitative data on the views and experiences of participants to understand the project’s impact.

Interviews are more useful where an individual perspective is required.

Focus groups are more useful where a group perspective is required. It is also particularly useful where participants in the group are less likely to agree to a solo interview (e.g. children).

Focus groups can also be pragmatic, gathering a wide range of views at once, or opportunistically linked to group activities (e.g. staff meetings).

Allows for the collection of detailed information on individual or collective impact of the project.

Can be organic, often engendering candid and authentic responses.

Offers further explanation for data already collected.

Might be more time intensive than other forms of data collection.

Risk of bias: certain individuals may be more or less willing/able to participate.

Focus groups require careful moderation in order to ensure all participants have equal opportinity to contribute.

Click below to view an interview and a focus group template that can be exported to a Word document.

Example interview template

Interviewee: Age: Researcher: Date:

Introduction (Adapt as necessary to be contextually and culturally appropriate)

Hello, my name's __________ .

This week the town’s community development team is talking to people who have been involved in the ‘Together 4 Tea’ program to find out what their experience has been like. We’ll be using this information to work out how well the ‘Together 4 Tea’ program helps people in our community. It’s optional to participate in these conversations, and you can choose to stop the conversation at any time.

Are you happy to talk to me today about your experience of 'Together 4 Tea'?

If yes: Giving your name and age is optional, would you like to give that information today?

Could you tell me what it was like to be part of 'Together 4 Tea'?

Do you plan to keep meeting your 'Together 4 Tea' group so long as you're able to? And if so, why, or why not?

If you weren't taking part in 'Together 4 Tea', how do you think your life today would be different?

Do you have regular meetings with other people or groups in the community?

Have you previously participated in any group meet-ups like 'Together 4 Tea' pr something similar?

Is there anything you'd like to add about 'Together 4 Tea'?

Example focus group template

Facilitator: Group: Date:
Program: Location:
Facilitator introduction to the group; contextually and culturally appropriate.

What made you get involved with 'Together 4 Tea'?

Thinking about your expectations when you joined the program, has it been what you expected?

Has anything changed in your weekly routine since you started attending 'Together 4 Tea'?

Do you feel more connected to your community since you started attending 'Together 4 Tea'?

What is the most significant change you've experienced as a result of your participation in 'Together 4 Tea'?

"Data collection and analysis can take a long time, and this should be taken into account when planning interviews..."

Section 6: Meaningful Findings


6.1 Data analysis: How to make sense of results

Quantitative analysis

Quantitative analysis involves assessing numerical data and can involve statistical analysis. Descriptive statistics summarise data and includes frequency, averages (or measures of central tendency such as mean, median and mode) and measures of spread (range, standard deviation and variance).

Common tools are:

  • Frequency: a total count or proportion of the variables.
  • For example: ‘50% of people feel safer in their homes as a result of neighbourhood watch’.

  • Central tendency: mean (the average value), median (the middle value) and mode (the most frequent value).
  • For example: ‘On average, senior residents attended three out of five community events a year’.

  • Correlations: the extent to which one variable influences another.
  • For example: ‘Participation in the course goes up 20% when course fees go down by 10%’.

  • Cross-tabulations: the relationship between two sets of variables.
  • For example: ‘Women are more likely to report increased social connection through participation in a program than men’.

  • Regression analysis: examining the impact of the program controlling for factors other than the program itself.
  • For example: ‘What is the participation rate for women when you factor in the effect of child care responsibilities?’.

Qualitative analysis

Qualitative analysis involves making sense of words, either spoken or written. A common method of qualitative data analysis is thematic analysis, where the content of text is looked at to identify key themes and patterns, and how common they are (through counting the number of mentions, for example).

Thematic analysis involves not just counting the words in text, but also understanding both the obvious and implied ideas and meanings. Sometimes themes and patterns can be easily identified, but often they are less clear and must be drawn out. Themes can be coded, counted (frequencies), compared with other themes and compared across quantitative variables (e.g. the responses of women versus men).

Thematic analysis can be:

  • Inductive: the themes emerge from an examination of the data itself (ground up), without any preconceived categories.
  • Deductive: the themes are theory driven and analysis tends to be limited to predetermined categories.

Simple steps for thematic analysis are:

  1. Familiarisation of the data, often through repeated reading and examination.
  2. Generating initial codes for themes.
  3. Searching for and sorting the codes: some responses may fit within more than one code or theme.
  4. Reviewing, refining and defining the themes.
  5. Making conceptual sense of the themes and connections between them.

"It is also useful to release findings early in the evaluation, even if the data quality is not yet good, becuase this is a way to engage stakeholders in the evaluation and encourage participation..."

6.2 Assessment and evaluation: How to make judgements of worth and merit?

Returning to purpose and context

When analysing information, it is important to return to your purpose and context and revisit your original program objectives, evaluation questions and program logic.

For example, if the objective of a program was to enable people from a specific migrant community to get together and enjoy doing an art activity, there is no need to prove that the program also created lasting social support, social inclusion and led to employment for participants. If you have evidence this happened—great, that is then the icing on the cake but it is not an essential ingredient.

Attribution and contribution

An important part of analysing data for an outcome or impact evaluation is to think of attribution and contribution.

Attribution is when we can say with confidence that ‘doing X resulted in a change in Y’ (so without X, then Y would not have changed).

Contribution is when we can say with confidence that ‘doing X contributed to a change in Y’ (without X, Y would not have changed as much, or that it was because many things were working together that helped to change Y).

It can be difficult to determine contribution and attribution—especially in complex programs with high level social change goals—as it is hard to separate the influence of a program from other factors that influence social change.

e.g. Attribution and Contribution

For example, it may be possible to determine that a community transport program for elderly residents reduced their social isolation with a high degree of confidence.

It may, however, be harder to determine that children’s school attendance rates increased due to a breakfast program, when a mentor program and sport coaching program were also targeting attendance in the same cohort.

There are also external factors occurring at a societal level. For example, a household budgeting course reduced financial stress, but there were other interventions occurring at the same time (e.g. JobSeeker payment increases for the unemployed).

Ideally, we would want to undertake regression analysis or conduct a randomised controlled trial to develop estimates of the impact of a program controlling for confounding factors but such analyses are difficult to implement.

Bringing others into sense making

Making sense of complex information can sometimes be challenging. The data analysis phase may benefit from involving a wider group of stakeholders.

One approach is to hold a formal evaluation summit workshop to bring together stakeholders to present and discuss findings and preliminary interpretations. This provides an opportunity to see where there is consensus on the findings, where alternative interpretations are possible and where some aspects of the evaluation puzzle are still missing.

This type of ‘sense making’ session can be done with a small group (e.g. evaluation reference group) or a large group (e.g. community workshops). Bringing others into sense making can be particularly useful when trying to understand attribution or contribution, or when findings are not congruent—for example if data from different sources are telling different stories.

"It is also useful to release findings early in the evaluation, even if the data quality is not yet good, becuase this is a way to engage stakeholders in the evaluation and encourage participation..."

6.3 Sharing results: How are evaluation findings presented?

Generally, evaluations conducted by CDOs will feed into regular reporting processes, and evaluation findings can be presented in discrete written reports, verbal presentations, videos or audio recordings.

Presentation of findings can be ongoing, at interim periods or at the end of programs or activities. Interim reports can provide early indications of a program’s success, flag issues or highlight changes needed, for example, identifying the need for increased marketing. It is also useful to release findings early in the evaluation, even if the data quality is not yet good, because this is a way to engage stakeholders in the evaluation and encourage participation.

Reports usually have the following sections (although there can be variation between different reports):

  • Summary: a brief overview of key findings and conclusions.
  • Introduction: purpose, objectives, evaluation questions, background.
  • Evaluation findings: program design, any underpinning evidence or theory, program logic and theory of change and evaluation approach (e.g. process evaluation).
  • Methodology: methods of data collection and analysis.
  • Findings: summary findings, often structured around the program logic or evaluation questions.
  • Conclusion: summarises the major findings and answers the evaluation questions.
  • Recommendations: advice for decision-makers.
  • Appendix/Appendices: relevant supporting documentation, such as copies of the survey or interview questions.

Not all evaluation reports will include recommendations, with some reports stopping at key findings and conclusions.

See Toolkit: Reporting for further detail and templates.

Tips for presenting key findings

  1. Use pictures, graphs and tables to highlight the most interesting findings and add variation for the reader. Make sure this is balanced (don’t overdo it), with adequate commentary.
  2. Build a narrative. Use the analysis to tell a story that focuses on answering the evaluation questions.
  3. Be careful when using percentages on their own to report findings where there are few responses. For example, if only 10 people answered a question, say ‘five respondents (50%) agreed that the program increased their knowledge of recycling and one disagreed’, rather than ‘50% of respondents agreed that the program increased their knowledge of recycling’.
  4. Use quotes from qualitative responses to highlight key themes and ‘give a voice’ to the quantitative data.
  5. Font weight, colour, text boxes or panels can be used to highlight major findings so that they stand out.
  6. Protect anonymity, and where identities are or may be disclosed, ensure permission is received.

"It is also useful to release findings early in the evaluation, even if the data quality is not yet good, becuase this is a way to engage stakeholders in the evaluation and encourage participation..."

Existing Data Analysis

Figure 15: Existing data analysis approach

Figure 15

Analysing existing data

How to Use Pros Cons

An analysis of existing data can be used to quickly gather background information about what has or has not worked before in similar contexts. It can be used to conduct a needs analysis.

Requires little effort and can be cost effective.

Plenty of sources of data are typically available.

Can provide an initial basic understanding and context to pair with other analytical tools.

No control over data quality, and data collection points cannot be set.

Click below to view an example existing data analysis plan template that can be exported to a Word document.

Example existing data analysis plan template

Researcher: Date:
Aim of the research: Outline the aimes and objectives of the evaluation and what your research questions are.
Background/rationale: Provide an explanation of why the evaluation is being done and what is already known about the topic.
Evaluation design: Provide an overview of the design and methods you will use, including the number of samples you will review and the data analysis you will undertake.
Relate these back to the aims and objectives of your evaluation

"It is also useful to release findings early in the evaluation, even if the data quality is not yet good, becuase this is a way to engage stakeholders in the evaluation and encourage participation..."

Reporting: How and when?

The evaluation reporting schedule should be set out in the evaluation plan. This will generally fit within your local government’s existing reporting timeframes (i.e. monthly, quarterly, annual reports) however it may differ depending on external funding requirements, partnership projects etc.

Reporting the findings of an evaluation is a core component of its design, and has the potential to feed into continual improvement of programs with adjustment and developments at various program iterations. As such, the timing of evaluation, including interim and final reports, is strongly tied to the success of activities.

In planning your reporting, you should take into account the time required to collect and analyse data, compile results, prepare an initial draft report and have it reviewed by a peer, with adequate time remaining to write revisions.

Click below to view a reporting template and a final evaluation report checklist template that can be exported to Word documents.

Reporting template:

Title page

Project name


Contact details

Date (month and year of publication)

Executive summary

A high-level overview that sets out important findings and offers decision-makers key insights in an efficient format.


Outline the major sections of the report and the primary people involved in analysis and production of the report.


Provide any relevant context that might not be included in the introduction—such as rationale for the evaluation or who initiated it and why.

Research questions

State the research questions that informed your evaluation.

Methods used

Describe step-by-step the process used to complete the data analysis.


Outline the findings and any recommendations that arose from the evaluation.

Final evaluation report checklist:

Executive summary:

This section caters for those who will not read the entire report and pulls out the key details.

Clearly describe what was evaluated.

Outline why the evaluation was conducted.

Summarise the major findings.

Summarise the key recommendations.

Identify who the report is aimed at.

Describe any major constraints upon the evaluation and the context in which they arose.


This section gives context to the program and its rationale.

Provide an overview of the origins of the program.

Describe the aims and objectives of the program.

Describe the participants and staff in the program.

Evaluation framework:

This section explains how and why the evaluation was designed and conducted. It addresses what was in and out of scope, and sets out the methods, tools and any other relevant information.

Explain the purpose of the evaluation.

Describe the context within which the evaluation was undertaken.

State the evaluation team members.

Set out the evaluation design.

State the evaluation questions.

Describe data collection procedures and instruments used.

Outline any limitations experienced.

Evaluation findings:

This section addresses the results of your evaluation. Organising results by using the key evaluation questions as subheadings can be useful. Not all data collected needs to be presented in this section; some may be presented in an appendix.

Outline results according to each of the key evaluation questions.

Draw insights from data collection to answer the key evaluation questions being addressed.

This section is not only about presenting data, but also interpreting information and making a value judgement.

Use graphics were appropriate.

Conclusion and recommendations:

This section provides a high level outline of the success of the project and lessons learned, based on evaluation findings. Key recommendations that could be used to inform future policy or program work should be included in this section (as well as the executive summary).

Summarise the success of the project.

Outline the key recommendations.

Identify the potential uses for these recommendations.

Set out how the evaluation findings will be used.

"Reporting the findings of an evaluation is a core component of its design..."

Section 7: Evaluation Management


7.1 Ensuring the evaluation is representative

Engaging stakeholders

Stakeholders include internal and external staff and volunteers, community groups and agencies, partners, funders and, importantly, beneficiaries.

Participatory evaluation approaches include a broad range of internal and external stakeholders in the design and conduct of an evaluation. These approaches might actively look for ways to ‘centre the perspectives of beneficiaries’. For example, members of a community can be engaged to co-design the evaluation and, as peer researchers, to interview or survey other community members.

Engaging stakeholders can add value to all stages of the evaluation. They can, for example, be actively involved in:

  1. Contributing to program design and evaluation planning.
  2. Testing program assumptions and developing program logic.
  3. Contributing to and/or facilitating data collection.
  4. Supporting evaluation engagement strategies.
  5. Helping to understand the data collected and what it means.
  6. Disseminating and utilising findings.
  7. Fostering ‘champions’ for your activities/program.

Inclusion and diversity

Any valid program evaluation should seek to represent the views and experiences of the whole community within which the program operates. Consider selective sampling, and communication activities tailored for various groups.

Within local governments, existing networks can be tapped to support the inclusive design and delivery of evaluations. This can be from within local government departments, or might include local disability groups, Aboriginal and Torres Strait Islander communities, CALD communities, or representatives and advocates for young people and senior citizens.

Some questions to consider are:

  • Will the people sampled in the evaluation reasonably reflect the diversity of people in the community (for general community programs), or represent those who are expected to benefit from the program (for targeted programs)?
  • Are the evaluation methods accessible to everyone in the community? Will there be technology, language, physical or cultural barriers to participation?
  • Is there a trust barrier that might prevent some groups from participating? Are partnerships required in order to include hard to access groups?
  • Will there be particular barriers for some members of the community to participate in the evaluation?
  • Have you ensured that measurement tools are suitable for all stakeholders? For example, are survey or interview questions accessible, with wording that is easily understandable?

"Incorporating any evaluation into your day-to-day work can help inform continual improvement and deliver better outcomes across activities, services and programs."

7.2 Sourcing evaluation expertise: Who will do the evaluation?

Generally, you will design and implement evaluations in collaboration with your team and supervisor. However, sometimes your local government may engage an external evaluator to conduct an evaluation (or parts of it), where independence is important, or where specialised skills are required.

Evaluations can also be done with a mixture of external and internal ‘evaluators’. For example, a project may use an external evaluator at the beginning and at the end of a program while utilising internal evaluators to undertake monitoring and continuous learning loops.

Some evaluations include setting up a reference group to help guide the different stages of evaluation. Evaluation reference groups can add value to evaluations, increase engagement and build local capacity. The groups can be made up of a small number of internal staff members or include community members or community partners.

Context, budget, purpose, available resources and philosophical approaches can all influence who undertakes evaluation.

"Incorporating any evaluation into your day-to-day work can help inform continual improvement and deliver better outcomes across activities, services and programs."

7.3 Ethics: What do I need to consider to ensure an ethical evaluation?

Evaluations should be planned and conducted in a way that takes into consideration ethical principles and practice.

Common ethical principles to incorporate into evaluation planning include:

  1. The evaluation should be inclusive and representative of all interested persons and groups.
  2. The evaluation should be designed, conducted and reported in a manner that respects the rights, privacy and dignity of those affected by and contributing to the evaluation.
  3. Data collection should ensure the right methods for the right audience and context.
  4. Participation in evaluation should be based on informed consent, with the subject of the data collection understanding:
    • the purpose of the evaluation
    • what will be done with the findings
    • likely risks
    • how confidentially will be managed
    • the extent to which participation is voluntary
    • In the case of minors, informed consent must be sought from parents or guardians.

  5. As a general principle, evaluations should ‘do no harm’. When dealing with sensitive evaluations, you need to consider how to deal with possible disclosures and what to do if the evaluation methods cause distress for participants.
  6. It is important to anticipate the risk of such disclosure or harm, and develop protocols for identifying and responding. Areas of potential trauma should be avoided where possible, and mechanisms put in place to ensure support for participants and evaluators, if required.

  7. Reciprocity—participants providing information should reap some benefit. For example, evaluation findings (or an evaluation summary) should be made available to participants where possible.
  8. The findings of evaluations should be based on rigorous design, valid data collection and analysis, and objective judgements that are based on the information collected.
  9. The evaluation should be reported in a way that is fair and balanced.

Resources for Evaluations:

The Australian Evaluation Society has developed a number of resources for the ethical conduct of evaluations:

Code of Ethics for AES members

Guidelines for the Ethical Conduct of Evaluations

"Incorporating any evaluation into your day-to-day work can help inform continual improvement and deliver better outcomes across activities, services and programs."

7.4 Governance: Oversight structures to ensure a quality evaluation

Generally, your evaluation quality will be overseen through supervision processes and/or the project manager. Oversight helps ensure the quality of an evaluation and is particularly important when an evaluation is conducted internally.

Common approaches to governance of evaluation programs include:

  • establishing an Evaluation Working Group
  • embedding evaluation expertise on the Project Advisory Group
  • peer review of evaluation findings

Note: these approaches may not be workable for smaller scale programs, services or events.

For some more complex areas of evaluation, engaging a ‘lived experience’ perspective will also be important. Models that could be applied include:

  • employing a lived experience person on the evaluation team or project team, and actively asking for their input into instruments and methods.
  • establishing a Lived Experience Advisory Group to provide feedback at regular times.
  • engaging peer researchers with lived experience who can help with data collection and provide a lived experience perspective.

"Incorporating any evaluation into your day-to-day work can help inform continual improvement and deliver better outcomes across activities, services and programs."

7.5 Linking to strategic planning and reporting

Local governments build in outcomes and output measures through their Integrated Planning and Reporting Framework, a key obligation under the Local Government Act 1995. The aim of the Framework is to ensure community priorities are integrated into strategic planning.

Consideration of the broader strategic planning and reporting world in which you do your work—and aligning outcomes and measures with this—can add value in terms of a coordinated approach and shared understanding within and across teams.

Having said that, this should not be a barrier to just getting on and doing any level of evaluation. In fact, data collected through ground-up evaluation activities will often add richness and depth to our more abstract reporting processes.

Figure 8 below provides one way to think about the relationship between evaluation in your day-to-day activities and strategic planning and reporting.

Figure 8: The relationship between evaluation, daily activities, strategic planning and reporting

Figure 8

"Incorporating any evaluation into your day-to-day work can help inform continual improvement and deliver better outcomes across activities, services and programs."

Further Resources

Evaluation tools and frameworks:

Logic models:

Stakeholder engagement:

Evaluation design:

  • Australian Institute of Family Studies - Planning for evaluation: basic principles. Questions to consider and in-depth consideration of data collection methods and tools for designing your evaluation plan.

Templates and examples of evaluation plans:


Advanced reading:


Australasian Evaluation Society, Guidelines for the Ethical Conduct of Evaluations.

Better Evaluation (n.d.) Cost Benefit Analysis.

Better Evaluation (n.d.) Rainbow Framework.

Better Evaluation (n.d.) Randomised Controlled Trial.

Better Evaluation – Revised Evaluation Criteria Definitions and Principles for Use.

Craig G (2002) ‘Towards the Measurement of Empowerment: the Evaluation of Community Development’, Journal of the Community Development Society, 33(1).

Davidson EJ (2012) Actionable Evaluation Basics: Getting succinct answers to the most important questions, Real Evaluation Ltd, Smashwords Edition.

Knowlton L and Phillips C (2013) The Logic Model Guidebook. Better Strategies for Great Results. Second Edition. Sage Publications, as cited in Better Evaluation.

Markiewicz A (2014) Core Concepts in Developing Monitoring and Evaluation Frameworks. Anne Markiewicz and Associates.

Markiewicz A and Patrick I (2016) Developing Monitoring and Evaluation Frameworks, Sage Publications, Thousand Oaks, California.

McDavid JC et al. (2013) Program Evaluation and Performance Measurement: An Introduction to Practice (2nd ed). Sage Publications, Thousand Oaks, California.

NSW Government (2013) Evaluation Toolkit, Office of Premier and Cabinet.

Organisation for Economic Cooperation and Development. (2020). Better Criteria for Better Evaluation: Revised and Updated Evaluation Criteria.

Productivity Commission, A Guide to Evaluation under the Indigenous Evaluation Strategy, Draft (2020).

Social Value UK (5 May 2009) SROI Guide Impact Map - worked example.

Social Ventures Australia Consulting. (2012) Social Return on Investment Lessons Learned in Australia.

Taylor-Powell E, Jones L & Henert E (2003) Enhancing Program Performance with Logic Models. University of Wisconsin-Extension. p. 181.

Warner M (2001) ‘Building social capital: The role of local government’, Journal of Socio-Economics, 30(2), pp. 187–192.

Wissemann A (1992) as cited by the Centre for Family Research and Evaluation.