1.2 Project Background

This project was initiated by LG Professionals WA and supported by a number of West Australian local government authorities who worked with the Centre for Social Impact, University of Western Australia (CSI UWA) to develop this resource, an associated training program and web-based resources.

What we heard

Results from a Community Development Evaluation and Measurement Survey undertaken for this project, reflect a growing desire amongst CDOs and leadership teams to increase internal evaluation capacity.

Survey responses

• “I think there is a lot more interest in CDOs measuring impact than there was, say, 5 years ago. There is also a lot more talk around how that relates to community wellbeing which sparks that debate.”
• “There definitely seems to be growing interest in better capturing the impact of our work…”

However, CDOs and leaders also reported resource constraints which included time, money, skills, training and capacity to support this work, and other organisational or cultural barriers to evaluation. For example, while senior leaders do give ‘in principle’ support for evaluation, this is often not translated into practice, or does not help unlock the resources required to conduct evaluations. Respondents also noted that when evaluations were conducted, they were sometimes ineffective and/or the findings were not well utilised. This tells us that CDOs are likely to need to:

About this Framework

This Evaluation Framework is a tailored response to the needs of CDOs—a guide to evaluation basics that can be effectively used within a range of settings.

This resource is a step towards creating a more systematic approach to evaluation across the state-wide network of CDOs, enabling CDOs and managers to develop a shared language and understanding about what is possible when evaluating their work. Outside of this resource, and through ongoing training, it is hoped that CDOs can support one another as the sector builds capacity, and that leaders can support this work too.

This resource consists of two parts:

• Part One: An Evaluation Framework providing:
• An overview of key things to consider when planning and managing an evaluation.
• A foundational approach to undertaking evaluations for community development purposes in local government settings.
• A discussion of main approaches to evaluation and methods that might be most suitable to a real-world local government context.
• Part Two: An Evaluation Toolkit providing
• Guidance on implementing the evaluation cycle and using different evaluation tools.
• Templates and resources.

1.3 Developing the Framework: core principles

Community development work:
understanding the change we make

At the start of this project, the CSI UWA attended a Community Development Forum, where CDOs from local governments across WA were asked to summarise their work with one ‘change word’. The most popular word was ‘facilitate’, selected by one third of the 65 respondents (32.3%), followed by ‘enable’ (18.4%). Interestingly, a quick glance at all the words suggested by CDOs at the Forum reveals a strong pattern: most change created through community development work often involves a mediator, and is one step removed from the ultimate impact point.

This tells us something important about the change created by CDOs. Firstly, the impacts are created in relationship with others, and secondly, the effects of community development work are more indirect than direct. In other words, CDOs effect change indirectly, and through working with others to support them to effect change.

Figure 1: Words to capture the nature of change created by CDOs

Processes, social capital and outcomes

This aligns with literature on community development which proposes that CDOs often create social capital, and the conditions of empowerment and agency. ¹

Phillips and Pittman (2009) note that practitioners of community development think in terms of outcomes (i.e. the “physical, social, and economic improvement in a community”), while most academics think of community development as a process or “the ability of communities to act collectively and enhancing the ability to do so”.

• Processes of community development are patterns of action that lead to greater empowerment of local communities.
• Social capital, or the ability to act, manifests as strengthened networks of people who feel empowered to act together more effectively to pursue their collective interests through facilitating, enabling and building capacity. ²
• Outcomes of community development can include community groups taking action, as well as specific improvements in the community. For example, more inclusive social networks. ³

It can be useful to think of the relationship between the process and outcome as in the ‘community development chain’ illustrated below.

The process, or the way change happens, can be equally, or even more important, than the outcome. For example, programs which achieve an improvement in health indicators, but are culturally inappropriate, deliver outcome goals but would fail an evaluation of process goals. ⁵

Evaluating outcomes alone overlooks the importance of empowerment and participation in community development initiatives. From this perspective, it is not whether or not the outcomes are achieved that matters, it is how they are achieved that is important.

Figure 2: Community Development Chain (from Phillips and Pittman) ⁴

Implications for evaluating community development work

To understand the impact of community development programs, it is important to evaluate the state of all three components of the community development model.

At first glance this may make evaluating outcomes seem less straightforward. However, evaluation is a highly adaptable process, and what this change model means for evaluation is that there should be a focus on the causal chain of events, not just the outcomes (i.e. use of program logic). Process evaluation will be just as important as outcomes or impact evaluation, and that evaluations should be done in partnership with stakeholders—as the relationships that CDOs engage are part of the change process.

Figure 3: Community Development Chain and Economic Chain ⁶

This Framework will:

1. Outline the use of program logic as a foundational tool for demonstration of impact.
2. Outline the importance of process evaluation, as well as impact evaluation, for demonstrating the impact of community development work.

2.2 Scoping Evaluations: What are the key questions?

Many systems that organisations set up around reporting are often based on rigid templates or pre-set targets and indicators. Evaluation is different and there is scope for it to be a much more open process. Evaluation is a pragmatic activity that aims to answer questions that stakeholders consider will be most useful to know, at a given point in time.

Evaluation relies on both quantitative and qualitative measurement to provide the information to answer the questions. Agreeing on a set of questions with other stakeholders normally forms the basis and starting point of an evaluation.

2.3 The evaluation cycle: What are the main evaluation phases?

Ideally, evaluation is built into the whole program cycle—from identifying what is needed through to designing an activity, planning, implementing, reflecting on results and determining program improvements. Importantly, good evaluation practice is to put the evaluation design in place before the program begins so that measurement occurs before, during and after the program has finished.

Figure 4 shows the interaction between cycles of evaluation and program development.

Figure 4: Evaluation in the program lifecycle

2.4 Planning: What needs to be considered at the start?

Steps in conducting an evaluation, at a basic level, are:

1. Clarify the evaluation focus and purpose—establish evaluation questions and a program logic.
2. Plan the process for communicating about the evaluation and engaging stakeholders.
3. Select which tools (e.g. surveys or interviews) and measures (the way in which we operationalise outcomes) will be used for collecting data.
4. Collect the data—before, during and after the program has ended.
5. Analyse the data—make sense of what it is saying and draw conclusions.
6. Answer the evaluation questions and report findings

Prior to getting started, it is good practice to create an evaluation plan.

Developing an evaluation plan

An evaluation plan details the approach to an evaluation, including how evaluation results will be utilised to improve the program and make decisions going forward.

Planning an evaluation in a local government setting should include:

1. Program logic (what is the change expected from program activities).
2. Outcomes, indicators and measures (what difference will the program make, what can you measure to know you have made a difference).
3. Measurement plan (when and how will you measure it).
4. Data collection plan (with tools such as a survey or interview schedule).
5. A list of stakeholders (including, importantly, the beneficiaries) and a plan to engage them at all stages.

2.5 Establishing purpose: What do we hope to find out?

Establishing the purpose is key to guiding the evaluation process.

Consider both the activity and the strategic context

The purpose will depend on both the nature of the activity and strategic factors.

1. The activities that you are likely to evaluate may include community events, programs, projects, place building initiatives, social plans, or recreation services to name a few. Just as the purpose of each of these activities varies, so too will the purpose of evaluating them.
2. What is happening in the wider strategic context that will inform the evaluation? Is there a need to focus on accountability, value for money, creating an evidence base for effectiveness?

Here are some practical examples:

• A community garden program is popular with residents but does not have a secure funding base. The evaluation may then focus on community outcomes and whether resources are being used effectively, to convince decision-makers of its worth to communities, and to ensure it continues in future.
• A play-based community parenting program has a strong, existing evidence base and there is a desire to implement it more widely. Due to the strong evidence already existing, the evaluation might not need to focus on outcomes as much as exploring implementation learnings before it is scaled up and rolled out in regional areas.
• Moving support for volunteering from a physical hub location to a decentralised, online model was thought to be an innovative approach. A pilot program is trialled to gauge effectiveness. The evaluation will focus on measuring success and outcomes when compared to the physical hub model, and document processes and costs so as to compare both models and make decisions about how to progress in future.
• A new program targeting participants from a vulnerable community focussed on employment pathways. This evaluation could employ participatory evaluation methods, allowing participants to say if they felt the program meets their needs and why. Asking questions about who did not engage and why not might also be important.

Develop your evaluation questions

It is important to ensure that your evaluation questions are linked to the purpose, that they are sharply targeted and well scoped.

Generally, evaluations have no more than five questions, although there may be sub‑questions.

Some examples of evaluation questions are outlined below: ⁸

• Relevance - 'Is the activity doing the right things?'
• Effectiveness - 'Is the activity achieving its objectives?'
• Efficiency - 'How well are resources being used?
• Impact - 'What difference does the activity make?'
• Sustainability - 'Will the benefits last?'

CDOs, leadership teams and community members may be involved in scoping the evaluation questions to ensure that all stakeholders will get some value from the information gleaned, and there are no important unanswered questions at the end.

3.2 How to build a program logic

A program logic can demonstrate, through causal links, that particular outcomes are likely to be achieved even if you are unable to measure all outcomes or long-term outcomes (which can be difficult to measure).

Developing a program logic will also help to identify what to measure and when. This can help to clarify what information you need to provide evidence for.

Program logic is best developed in a team that includes partners, stakeholders and leaders. This will ensure it captures different perspectives and ideas. Ideally, a ‘co-design’ approach can be used which will also include beneficiaries and community representative/s. This approach can provide lived experience insights which may help clarify or correct any assumptions.

A review of evidence to validate the causal links and assumptions in your program logic can significantly strengthen your logic model.

This evidence review can consider different fields of knowledge, or different lenses. For example, you can validate links using what you know from documented lived experience perspectives, a scientific lens (using for example systematic reviews of the evidence) and/or a policy lens. In many cases these multiple lenses will reinforce one another.

e.g. Building program logic

A school-based breakfast program aims to improve educational outcomes for at-risk students. While it may seem a stretch to suggest a simple breakfast program with minimal funding can achieve educational outcomes, these causal links can be mapped using a program logic and validated through existing knowledge.

In the above example, you could consult with participants (lived experience) to validate whether it is true that the chance to have breakfast at school does motivate them to attend. Then, you could validate the link between increased attendance and improved literacy and numeracy outcomes through a literature search. Links between attendance and improved educational outcomes are also likely to be validated and politically endorsed in the policy literature and key frameworks available in the education sector.

This maps a pathway between cause and effect, to help determine whether progress is as expected and, if not, what might be affecting it. It is beneficial to return to your program logic during the implementation process. It is a useful tool to reflect with others upon any changes that have occurred.

Closely related to program logic is a theory of change. While a program logic visually sets out the different elements of a program and establishes their links, a theory of change is a statement which explains the rationale or reason for the linkages between elements of the program logic and the expected change. A logic model describes, while a theory of change explains. With both program logics and theories of change there may be great variation in the extent to which underpinning evidence already exists. In some cases the theory or logic may as yet be untested. In other cases the theory is well established, and the theory of change will make this explicit.

3.3 Identifying what information you need or "operationalising the program logic"

Once a program logic has been developed, identifying what to measure will be easier.

Performance information should be sought across all parts of the program logic including inputs, outputs, short-term or immediate outcomes, and if possible, longer‑term outcomes and impacts too. This is sometimes referred to as ‘operationalising’ the program logic.

In this process, the theoretical concepts within your program logic (e.g. eating a good breakfast will result in improved engagement with school) are turned into specific variables that you can measure (e.g. increased attendance rate) or for which you can even set a target (e.g. a 50% increase in school attendance or a 15% improvement in school performance). Data may be from existing data sources or data collection that occurs as part of your evaluation, or a mix of both.

Identifying data sources

A practical way to begin is by identifying what information is already being collected. Input and output data will be easy to access and collect as part of the management and administration of the activity. For example, project officers should be collecting records of room hire, number of participants who attended the workshop, number of workshops held, as part and parcel of their management of the program.

Outcome data may require more dedicated data collection efforts. Within your program logic you may have developed outcome statements such as: “students are more engaged in school as a result of attending the program”. A good indicator for this might be attendance rates which are already collected by schools. This is a very reliable data source and should be easily accessible.

However, for many immediate and longer-term outcomes there is often a need for planned data collection.

Outcomes measurement asks questions about change in a way that attempts to link the changes being measured to the program itself. You may ask participants directly, for example, what they have gained from the workshops. Questions asked might include: What has changed in the lives of individuals, families, organisations, or the community as a result of this program? Has this program made a difference? ¹¹

e.g. Operationalising the program logic

Operationalising the program logic takes outcome statements (such as increased engagement in community centre programs) and translates them into context-specific indicators (such as a 50% increase in enrolments) and measures (e.g. registration forms). A worked example of a parent education program (up to the point of determining appropriate measures) is below.

Figure 6: An operationalised program logic for a parent education program ¹²

Program Logic Template

Developing a clear program logic establishes a connection between the resources and inputs related to an activity, and the outcomes and impact it intends to achieve. When designing an evaluation, the program logic will help inform the data collection methods and key evaluation questions. It also provides an opportunity to identify external influences upon a program’s outcomes, as well as potential unintended consequences (both positive and negative).

There are many ways to set out a program logic, however the layout matters far less than the ability to clearly represent the cause-and-effect relationship between inputs, activities and outcomes, and whether the diagram makes sense to its intended audience.

Evaluation Learnings from the 'Gold Standard'

This section of the Toolkit provides an overview of the ‘gold standard’ of measurement—the randomised controlled trial—and an associated method, the quasi-experimental approach.

Community development work does not happen in a laboratory where factors can be isolated and controlled easily. The complexity of social and community interventions, the indirect effects and nature of the work, as well as resource constraints, will mean that there may not be opportunities to conduct meaningful quasi-experimental designs. However, this method provides important foundational understanding for research and evaluation practitioners.

Randomised Controlled Trial

To gather evidence for the effectiveness of an intervention, there is no better or more robust approach than the randomised controlled trial (RCT). An RCT measures the effect of an activity by measuring outcomes achieved, and then comparing that to a measure of what would have happened if the intervention had not been applied (i.e. the counterfactual).

In an RCT, two groups of randomly selected participants are separated into a control group and an experimental group. The experimental group receives the intervention while the control group does not.

The random selection process is an effective way to remove bias, especially in high sample sizes, to ensure the two groups start out with relatively the same composition or mix of characteristics. This means that outcomes in the experimental group can be more easily attributed to the intervention. If the sample has been randomly selected, it is possible to generalise the results to other populations (providing the sample is large enough).

This design represents the ‘gold standard’ approach—the approach demanded, for example, from peer-reviewed journals or medical research, where stakes are high and evidence needs to be especially robust.

Figure 9: Example randomised controlled trial framework (n = sample size)

Quasi-experimental Design

As social interventions are more complex and multidimensional than medical interventions, social research often involves an adapted form of the RCT, known as a quasi-experimental approach. A quasi-experimental approach acknowledges that a pure RCT is not possible in social contexts, but will still draw upon a similar design (through the use of a control group or control data) to find evidence of impact.

To successfully implement a quasi-experimental design, a specific set of conditions is required which include:

• program design and evaluation design occur simultaneously (at the start)
• the ability to capture baseline data (or pre/post data) to measure change over time
• access to control data, or a control group
• resources to engage and collect data from a control group
• adequate sample size of intervention participants and control group participants
• enough time for change to be realised, and for evaluation methods to capture this change.

Measuring differential impact: Control groups and control data in social settings

The problem to resolve when researching in social settings is how to create a control group of randomly selected participants? It would be unethical to recruit participants for a program and then deliberately deny them the chance to participate, especially if that program meets an important need.

In community programs participants mostly self-select into programs, which means the participants might have different characteristics to non-participants in ways that will affect outcomes.

One approach to resolving this is the ‘difference in differences’ method (also known as the DID or DD method). In this method, we compare how much each group changes over a period of time with respect to a certain outcome and then compare the extent of the change between the two groups.

Where it is not possible to access a control group, or where there are no resources to measure outcomes across a control group, it may still be possible to use control data—for instance, by using published population-level statistics relating to the outcome you are measuring.

In this way, it is possible to determine how the outcomes achieved by the initiative compare to the counterfactual (the outcomes achieved by doing nothing).

e.g. An example of the difference in differences method

Let’s say you are running a program to reduce rates of smoking in your community. How can you measure that rates did decline and, in particular, rates of smoking for people participating in your program (to demonstrate program effectiveness)? It will be easy enough to collect data from program participants that may show smoking rates decreasing. But without a control group or control data we do not know if this outcome was because of our intervention or due to other factors.

Societies are complex, and many factors impact upon outcomes. For example, in Australia, smoking cigarettes sharply decreased over a period of time, as a result of many different factors including public health campaigns targeting children to prevent smoking, GPs encouraging people to quit smoking, rising costs of smoking, and an increased interest in healthy lifestyles.

The following graph uses fictional data and a fictional ‘quit smoking’ program to demonstrate the ‘difference in differences’ method that can be applied in such a case.

Figure 10: Rates of smoking over time in program participants versus total population (illustration only)

Using this approach, we can see that those who received the intervention appeared to have reduced smoking at a greater rate than those in the total population—giving strength to the claim that the intervention was effective. Thus, the difference in differences method can demonstrate the measurable effectiveness of the program.

Summary

Whilst these evaluation designs will provide the most robust evidence, in reality, evaluators must be pragmatic. Effectively measuring change over time in an intervention group, as compared to a control group (thus measuring the differential impact), is very rarely implemented fully in a social context.

As one Community Development Measurement and Evaluation survey respondent commented, “Evaluation is identified as important by most of the team but it’s impossible to incorporate and complete a comprehensive, relevant, meaningful, evaluation process with inadequate resourcing and processes which then allow for linking evaluation recommendations to planning future work with the community.

A realistic approach can involve a mix of methods which provide enough quality information to inform decisions, even if the evidence is not strong enough to be generalisable or to be published in a peer-reviewed journal.

Planning an Evaluation

Steps in evaluation planning

There are many ways to approach evaluation planning. One example, adapted from the NSW Government Evaluation Toolkit, is set out below, and includes seven key steps.

1. Create a program logic.
2. Develop the evaluation plan.
3. Assemble the evaluation team or commission a consultant.
4. Develop the evaluation design.
5. Develop the evaluation work plan.
6. Manage the implementation of the work plan.
7. Disseminate evaluation findings.

What to include in an evaluation plan

The following provides a list of what things you should consider including in an evaluation plan.

About the program/activity

• Title
• Alignment to Strategic Community Plan/Corporate Business Plan
• Location/s
• Team Members
• Lead department/team
• Key stakeholders and beneficiaries
• Objectives
• Program logic
• Budget (including evaluation)

Framing the evaluation

• Purpose and focus of the evaluation: the 'why'
• Evaluation questions: what in the strategic context is important, what questions are you trying to answer?
• Evaluating approach: process or outcomes focus
• Monitoring requirements
• Considerations: what particular considerations led to the chosen approach?

Resource requirements

• Personnel
• Budgets (also consider budget to reimburse participants involved in lengthy interviews or focus groups for their time in this section)
• Data storage and collection systems

Potential risks and ethical issues and how they will be managed

• Who are the vulnerable groups or individuals who might be impacted by the work?
• What unintended consequences of the work could arise?
• What might go wrong at different stages of the work?
• What might prevent the project from being delivered on time and within budget?
• What negative impacts could the work have upon different groups?

Analysis and synthesis

• What standards will be used to assess merit (are there targets or benchmarks already set, will you make judgements based on sector standards, or criteria co-designed with lived experience advisors)?
• Who will be involved in analysing data and reaching conclusions?
• How and when will analysis happen?

Templates for planning

Using table-based templates can provide a quick way to capture the key information required within an evaluation plan and can also be used to set up reporting and work plans. The following are template resources for an evaluation plan, reporting plan, and work plan with example entries based on ‘Together 4 Tea’

5.2 Data collection: How do we measure it?

Once you have reviewed your program logic (and evaluation questions), you will have a better idea of what existing data you want to access and use. You may also decide that you need to collect further information as part of the evaluation.

Analysis of existing data and evidence

Data and evidence can be obtained from sources such as project documents, government records and publicly available statistics. The Campbell Systematic Reviews (Campbell Collaboration) and Cochrane Library provide access to systemic reviews, which can be useful sources of knowledge.

Common data collection methods

A summary of some methods to choose from is provided here.

Document reviews

Data and evidence can be collected from existing documents for the purpose of background information.

Case studies

A case study is a detailed investigation of a person, group or event in its real-world context, over a period of time.

Surveys

Almost all written surveys are now conducted via web-based platforms such as Survey Monkey, which you can access for free.

Surveys are convenient and can be anonymous.

When surveys are used to gather information from a large number of people, they can provide statistical findings that may be generalised to the broader population.

When undertaken at various intervals (e.g. prior to, during and after program implementation) the data can capture change over time.

Surveys give limited opportunity to gather in-depth information, and low response rates and small sample sizes will limit your ability to generalise findings, or do meaningful quantitative analysis. (As a very general guide, generally aim for a sample size of 30 people or more to make a survey worth doing.)

Interviews

Interviews can be conducted in person, by telephone or online. Consider which setting will best ensure respondents are comfortable sharing information. The mutual sharing that happens in an interview encourages the building of trust and can facilitate information gathering from sources who may otherwise be difficult to access.

In-depth discussions can uncover deep insights and unique perspectives and, as such, interviews can help understand why and how an intervention has worked. Interviews are suitable for use at any stage prior to, during, or after a program.

While in-depth views and opinions are easily accessed through this method, gathering large quantities of data through interviews can be resource intensive. Data collection and analysis can take a long time, and this should be taken into account when planning interviews.

Focus groups

Focus groups are an informal way of interviewing groups of typically 6–12 people, and usually last for around 90 minutes. They allow people to express their opinions and ideas freely and encourage open expression due to dynamic conversation created through multiple perspectives. Group discussion is rich and can complement or build on quantitative survey. Focus group members can be presented with results to be interpreted and emerging patterns to be explored.

Focus groups can help build understanding of why and how an intervention has worked or might work, and are suitable for use at any stage prior to, during, or after a program.

Given the short timeframe and nature of focus group discussions, there is little opportunity to cover many topics in this setting. A broader range of ideas can be discussed in a one-on-one interview setting.

Selecting a method

A number of factors will impact upon the types of methods used to conduct an evaluation, including:

• The stage of the program being evaluated:
• Is the program already underway? This gives you scope to access participants with some degree of ease.
• Is it already complete? Consider how you will access participants and how many might respond to a request for data collection.
• Is it about to enter the planning and design phase? This gives you the greatest scope to design an evaluation that will provide the strongest evidence on impact and implementation. Designing your evaluation prior to the start of the program is the preferred approach as it gives you the best chance to establish pre-post data and develop comparison reference points.
• Timeframe:
• Is there scope to capture data at multiple points in time?
• How quickly does the evaluation need to be developed in order to access participants during program delivery? Methods such as interviews can be more time consuming, while surveys can be turned around faster.
• If the program operates in cycles, data collection can be designed around this, and feed into iterations of the program.
• Stakeholders:
• Are there accessibility considerations which might impact respondents’ ability to participate? Certain formats may be more appropriate in certain instances. For example, interviews might be more culturally sensitive in Indigenous communities when approached in an appropriate manner. Include a broad set of stakeholders in your evaluation planning and always include a role for beneficiaries of programs.
• The type of data sought:
• A mixed methods approach is ideal as both quantitative and qualitative data will provide more robust evaluation findings.
• Prior evaluation experience:
• It is okay to select methods based on what you feel most comfortable using. A ‘good enough’ approach is accepted by expert evaluators: real world constraints will always be present. Start simply and build capacity with experience, while knowing that lack of prior evaluation experience is not a barrier to conducting an evaluation.

How to find and assess existing data

Assessing existing measures - questions to ask:

• How reliable and valid is the data source?
• Is it fit for purpose?
• Do you need to translate it to Australian or other context?
• Is the measure well-suited to the target population?
• Is it culturally appropriate and relevant?
• Are staff members qualified and able to administer?

Methods for collecting original data

Surveys

Click below to view an example survey template that can be exported as a Word document.

Interviews and Focus Groups

Click below to view an interview and a focus group template that can be exported to a Word document.

6.2 Assessment and evaluation: How to make judgements of worth and merit?

Returning to purpose and context

When analysing information, it is important to return to your purpose and context and revisit your original program objectives, evaluation questions and program logic.

For example, if the objective of a program was to enable people from a specific migrant community to get together and enjoy doing an art activity, there is no need to prove that the program also created lasting social support, social inclusion and led to employment for participants. If you have evidence this happened—great, that is then the icing on the cake but it is not an essential ingredient.

Attribution and contribution

An important part of analysing data for an outcome or impact evaluation is to think of attribution and contribution.

Attribution is when we can say with confidence that ‘doing X resulted in a change in Y’ (so without X, then Y would not have changed).

Contribution is when we can say with confidence that ‘doing X contributed to a change in Y’ (without X, Y would not have changed as much, or that it was because many things were working together that helped to change Y).

There are also external factors occurring at a societal level. For example, a household budgeting course reduced financial stress, but there were other interventions occurring at the same time (e.g. JobSeeker payment increases for the unemployed).

Ideally, we would want to undertake regression analysis or conduct a randomised controlled trial to develop estimates of the impact of a program controlling for confounding factors but such analyses are difficult to implement.

Bringing others into sense making

Making sense of complex information can sometimes be challenging. The data analysis phase may benefit from involving a wider group of stakeholders.

One approach is to hold a formal evaluation summit workshop to bring together stakeholders to present and discuss findings and preliminary interpretations. This provides an opportunity to see where there is consensus on the findings, where alternative interpretations are possible and where some aspects of the evaluation puzzle are still missing.

This type of ‘sense making’ session can be done with a small group (e.g. evaluation reference group) or a large group (e.g. community workshops). Bringing others into sense making can be particularly useful when trying to understand attribution or contribution, or when findings are not congruent—for example if data from different sources are telling different stories.

6.3 Sharing results: How are evaluation findings presented?

Generally, evaluations conducted by CDOs will feed into regular reporting processes, and evaluation findings can be presented in discrete written reports, verbal presentations, videos or audio recordings.

Presentation of findings can be ongoing, at interim periods or at the end of programs or activities. Interim reports can provide early indications of a program’s success, flag issues or highlight changes needed, for example, identifying the need for increased marketing. It is also useful to release findings early in the evaluation, even if the data quality is not yet good, because this is a way to engage stakeholders in the evaluation and encourage participation.

Reports usually have the following sections (although there can be variation between different reports):

• Summary: a brief overview of key findings and conclusions.
• Introduction: purpose, objectives, evaluation questions, background.
• Evaluation findings: program design, any underpinning evidence or theory, program logic and theory of change and evaluation approach (e.g. process evaluation).
• Methodology: methods of data collection and analysis.
• Findings: summary findings, often structured around the program logic or evaluation questions.
• Conclusion: summarises the major findings and answers the evaluation questions.
• Recommendations: advice for decision-makers.
• Appendix/Appendices: relevant supporting documentation, such as copies of the survey or interview questions.

Not all evaluation reports will include recommendations, with some reports stopping at key findings and conclusions.

Tips for presenting key findings

1. Use pictures, graphs and tables to highlight the most interesting findings and add variation for the reader. Make sure this is balanced (don’t overdo it), with adequate commentary.
2. Build a narrative. Use the analysis to tell a story that focuses on answering the evaluation questions.
3. Be careful when using percentages on their own to report findings where there are few responses. For example, if only 10 people answered a question, say ‘five respondents (50%) agreed that the program increased their knowledge of recycling and one disagreed’, rather than ‘50% of respondents agreed that the program increased their knowledge of recycling’.
4. Use quotes from qualitative responses to highlight key themes and ‘give a voice’ to the quantitative data.
5. Font weight, colour, text boxes or panels can be used to highlight major findings so that they stand out.
6. Protect anonymity, and where identities are or may be disclosed, ensure permission is received.

Existing Data Analysis

Figure 15: Existing data analysis approach

Analysing existing data

Click below to view an example existing data analysis plan template that can be exported to a Word document.

Reporting: How and when?

The evaluation reporting schedule should be set out in the evaluation plan. This will generally fit within your local government’s existing reporting timeframes (i.e. monthly, quarterly, annual reports) however it may differ depending on external funding requirements, partnership projects etc.

Reporting the findings of an evaluation is a core component of its design, and has the potential to feed into continual improvement of programs with adjustment and developments at various program iterations. As such, the timing of evaluation, including interim and final reports, is strongly tied to the success of activities.

In planning your reporting, you should take into account the time required to collect and analyse data, compile results, prepare an initial draft report and have it reviewed by a peer, with adequate time remaining to write revisions.

Click below to view a reporting template and a final evaluation report checklist template that can be exported to Word documents.

7.2 Sourcing evaluation expertise: Who will do the evaluation?

Generally, you will design and implement evaluations in collaboration with your team and supervisor. However, sometimes your local government may engage an external evaluator to conduct an evaluation (or parts of it), where independence is important, or where specialised skills are required.

Evaluations can also be done with a mixture of external and internal ‘evaluators’. For example, a project may use an external evaluator at the beginning and at the end of a program while utilising internal evaluators to undertake monitoring and continuous learning loops.

Some evaluations include setting up a reference group to help guide the different stages of evaluation. Evaluation reference groups can add value to evaluations, increase engagement and build local capacity. The groups can be made up of a small number of internal staff members or include community members or community partners.

Context, budget, purpose, available resources and philosophical approaches can all influence who undertakes evaluation.

7.3 Ethics: What do I need to consider to ensure an ethical evaluation?

Evaluations should be planned and conducted in a way that takes into consideration ethical principles and practice.

Common ethical principles to incorporate into evaluation planning include:

1. The evaluation should be inclusive and representative of all interested persons and groups.
2. The evaluation should be designed, conducted and reported in a manner that respects the rights, privacy and dignity of those affected by and contributing to the evaluation.
3. Data collection should ensure the right methods for the right audience and context.
4. Participation in evaluation should be based on informed consent, with the subject of the data collection understanding:
• the purpose of the evaluation
• what will be done with the findings
• likely risks
• how confidentially will be managed
• the extent to which participation is voluntary

In the case of minors, informed consent must be sought from parents or guardians.

5. As a general principle, evaluations should ‘do no harm’. When dealing with sensitive evaluations, you need to consider how to deal with possible disclosures and what to do if the evaluation methods cause distress for participants.

It is important to anticipate the risk of such disclosure or harm, and develop protocols for identifying and responding. Areas of potential trauma should be avoided where possible, and mechanisms put in place to ensure support for participants and evaluators, if required.

6. Reciprocity—participants providing information should reap some benefit. For example, evaluation findings (or an evaluation summary) should be made available to participants where possible.
7. The findings of evaluations should be based on rigorous design, valid data collection and analysis, and objective judgements that are based on the information collected.
8. The evaluation should be reported in a way that is fair and balanced.

7.4 Governance: Oversight structures to ensure a quality evaluation

Generally, your evaluation quality will be overseen through supervision processes and/or the project manager. Oversight helps ensure the quality of an evaluation and is particularly important when an evaluation is conducted internally.

For some more complex areas of evaluation, engaging a ‘lived experience’ perspective will also be important. Models that could be applied include:

• employing a lived experience person on the evaluation team or project team, and actively asking for their input into instruments and methods.
• establishing a Lived Experience Advisory Group to provide feedback at regular times.
• engaging peer researchers with lived experience who can help with data collection and provide a lived experience perspective.

7.5 Linking to strategic planning and reporting

Local governments build in outcomes and output measures through their Integrated Planning and Reporting Framework, a key obligation under the Local Government Act 1995. The aim of the Framework is to ensure community priorities are integrated into strategic planning.

Consideration of the broader strategic planning and reporting world in which you do your work—and aligning outcomes and measures with this—can add value in terms of a coordinated approach and shared understanding within and across teams.

Having said that, this should not be a barrier to just getting on and doing any level of evaluation. In fact, data collected through ground-up evaluation activities will often add richness and depth to our more abstract reporting processes.

Figure 8 below provides one way to think about the relationship between evaluation in your day-to-day activities and strategic planning and reporting.

Figure 8: The relationship between evaluation, daily activities, strategic planning and reporting

Further Resources

Evaluation tools and frameworks:

• Arts Queensland resources on evaluation - An easy-to-read resources from the Queensland government, designed for the Australian context, particularly targeted at arts and culture.
• Technical Guide for Outcomes Measurement, NSW Office of Social Impact Investment.
• Public health and wellbeing outcomes measurement framework, produced by the Victorian government.
• The Evaluation Toolbox has a short tutorial on developing a Monitoring and Evaluation plan.
• BetterEvaluation's Rainbow Framework is a comprehensive and easy to use evaluation planning tool. It covers evaluation tasks and options from setting up an evaluation to reporting and utilisation of findings.

Logic models:

• Kellogg Foundation Logic Model Development Guide - An extensive, 'go to' resource for developing and understanding logic models, including templates.
• The University of Wisconsin-Madison Program Development and Evaluation site.

Stakeholder engagement:

• Cancer Australia 'Consumer Involvement Toolkit' contains multiple resources on consumer involvement, including service delivery and research. The knowledge presented is applied, accessible and transferrable. Contains checklists and templates.
• Preskill, H. & Jones, N. 2009. "A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions." Robert Wood Johnson Foundation. This resource includes strategies and checklists for engaging stakeholders.
• Bennett, S, Reeve, R., Muir, K., Marjolin, A., Powell, A. (2016), Orienting your journey: An approach for indicator assessment and selection, Toolkit, Sydney: Centre for Social Impact.

Evaluation design:

• Australian Institute of Family Studies - Planning for evaluation: basic principles. Questions to consider and in-depth consideration of data collection methods and tools for designing your evaluation plan.

Templates and examples of evaluation plans:

• Australian Institute of Family Studies - Planning an evaluation step by step.

Ethics:

• Australian Evaluation Society Ethical Guidelines including a Code of Ethics.
• Guidelines for Ethical Research in Australian Indigeneous Studies, Australian Institute of Aboriginal and Torres Strait Islander Studies, 2012. Working in a culturally safe way with Indigeneous Australians. Principles of ethical research with Indigeneous Australians, and how to apply them.
• The UWA Human Research Ethics Committee web page provides access to templates for consent and information sheets which you can adapt.

Advanced reading:

• Developed by HM Treasury, the Magenta Book series is a comprehensive guide to designing an evaluation framework.