You can see your service users achieving their goals. They are getting a job, keeping that job, getting healthy, finishing study or completing training, finding consistent housing, staying out of prison, connecting with the community, becoming more independent and feeling happier. Wow, this feels good!
You have watched them enter your service perhaps feeling vulnerable and isolated, and while pathways to recovery may differ, through the support of your organisation and their own dedication, their lives have significantly improved.
So, how do you begin to quantify this? And why?
Beyond the satisfaction of seeing positive outcomes and change, or knowing it is occurring as a result of your service’s interventions, quantifying change through outcomes measurement has fast become a requirement for the provision of public funding and grants for community and not-for-profit organisations. The landscape is changing, and as I noted in my recent blog: “Boards need to seek outcome data from management and hold the organisation accountable to deliver on its mission and outcomes.”
Measuring outcomes in a systematic, efficient and accurate way is now imperative. It can feel daunting, but it can be done.
What has been happening in the community sector?
The measurement of program and service outcomes has been steadily increasing in the community service sector. Regardless of size and internal capacity, organisations are improving in the collection, analysis, reporting and utilisation of data. Staff in these originations understand and embrace data capture and outcomes measurement practices as integral to their core business.
However, despite these strides, the practice of outcomes measurement within the community service sector reduced from 76% (2018) to 71% in 2019, across small, medium and large organisations. [i] Also, whilst an organisation can measure outcomes, how well they measure them is another (subjective) question. Last year, 75% of community organisations reported to “understand their outcomes”, but only 53% agreed that they measured their outcomes well, and even less so (35%) agreed that the community sector at large measured outcomes well.
Despite an increased interest and commitment to outcomes measurement, there is a lack of confidence in the actual practices being carried out internally as well as across the sector.
So, why measure outcomes?
There is a comforting level of transparency (particularly for the general public) when organisations provided with public funds identity what they want to achieve in the delivery of their services, and provide evidence that those outcomes have been attained.[ii].
But beyond this, there is a lot more to be said for improving outcomes measurement and reporting. Doing so provides community organisations with positive opportunities to:
Make more informed decisions for future service planning and provision, including refinements to models of care and resource allocation to drive further impact;
Secure further funding for ongoing or new services and programs which are demonstrated to be making positive changes as well as receive contracted outcomes-based performance payments for targets met;
Demonstrate the impact they make in the lives of individuals to the broader community, and in turn, increase philanthropic support;
Motivate the community workforce to continue the delivery of support to outcomes by providing objective feedback on results attained;
Foster innovative responses to program design weaknesses or contribute to the evidence base in confirming the effectiveness of particular programs; and
Facilitate collaboration and networking with other organisations to achieve an individual’s outcomes which may not be able to be met through one particular service or program.
What to measure
What is measured depends on three things:
What outcomes your programs or services are designed to achieve (Housing? Employment? Improved mental health?).
What can be realistically and sustainably collected and analysed? Considering both the metric (a job or housing placement is one thing, but employment or housing sustained for a length of time tells another more positive story) and the data collection method (quantitative or qualitative; there is a place for both, always).
How long you will measure that outcome: ideally, throughout the program, on exit from the service, and longitudinally to demonstrate impact.
While a mix of qualitative and quantitative should be collected and reported, organisations tend to mostly collect outcomes data qualitatively.
Case studies are a particular favourite …the presentation of a client’s journey to recovery can both pull at the heartstrings and provide a sobering insight into experiences different from your own.
A recent survey of the sector identified the most common approach to outcome measurement as case studies, client interviews and client satisfaction surveys (with 40-50% of organisations administering these methods more than once a year in 2019, and only 14% using standardised quantitative tools).[iii]
Best practice outcomes measurement
Start with culture change. Best practice outcomes measurement begins with culture change so that it is seen as standard practice integral to service delivery and not just a “nice to have”. [iv]To not only accept but embrace the practice for what opportunities it provides, and then adhere to it despite the increased workload.
Make it standard practice and cost and resource accordingly. Ensure some form of outcomes measurement and evaluation are included when tendering for any service, no matter how small, and commit between 5% and 20% of program costs to that end. Ensure you have sufficient staff to monitor and report on those outcomes.[v]
Identify all outcomes. Ensure that all of the outcomes that are required for an individual are identified. Whilst some of these outcomes may be beyond the remit of your program or service, good holistic care will result when you work with other relevant organisations who can assist. Underpinning the design of your program with a sound program logic or theory of change model, will help you account and plan for all necessary outcomes.[vi]
Use quantitative data. Ensure your outcomes measurement approach uses finely tuned quantitative measures, which are consistently measured in accordance with the technical design. Investing in IT is crucial when you are scaling up outcomes measurement.[vii]
Implement change! Data should not only be collected and analysed, it should be used. Identify why outcomes have not been achieved as intended, why they may have reduced over time, why some sub-groups within your target group are achieving outcomes and some are not. Then make the necessary changes to improve the program or service in response to that data.
Six steps to get started
There are small steps which can be taken to start measuring outcomes, or to take practice to the next level if it has stalled. Here are some tips to get your organisation started, or moving forward again:
Tip 1. Step beyond the boundaries of a competitive tendering environment and engage with other similarly sized organisations about how they measure their outcomes. Ask what they learnt, how they did it, what they measured, and what they would do differently.
Tip 2. Begin by looking deeper into a program which already has existing data collection infrastructure in place.[viii] Similarly, rely on validated industry measurement tools rather than developing your own internal outcomes measurement system (this can come later).
Tip 3. Identify one program, or a new program, as a focus point for improving outcomes measurement. Use this as an exemplar of improved measurement from which learnings, refinements and culture change can grow. Promote this program within your organisation as evidence that sound outcomes measurement can be achieved at this smaller level before scaling up from there.
Tip 4. Make it standard practice to include outcomes measurement in tendering for services and if outcomes are defined by the funder, go further by recommending the measurement of outcomes which may have been overlooked and how you will provide this data.
Tip 5. Get involved in workshops and framework development (often run by Government departments) to further understand outcomes measurement, build capacity and have a say in which outcomes should be measured in your sector, and how.
Tip 6. If you are further along the outcomes measurement journey, look at your organisation’s programs and services to see where outcomes can be measured and tools used or collated across the board to demonstrate a higher scale impact at an organizational rather than at the program or service level.
What’s next?
Once these practices are established and embedded (and not underestimating that this alone takes significant financial and resourcing investment), there are opportunities to take this measurement beyond program by program measurement to aggregate similarly defined outcomes across programs and services to provide a whole-of-organisation view of achievements and impact. Further still, economic analysis to demonstrate financial impact derived from outcomes achieved can add to the story of the value of a community organisation to the public and to the Government.
References
[i] Callis, Z., Seivwright, A., & Flatau, P. (2019) Outcomes Measurement in the Australian Community Sector: A National Report Card, Bankwest Foundation Social Impact Series, No. 10, Bankwest Foundation, Western Australia. [ii] Salesforce.org (n.d) Non-Profit Trends Report: Trends, Challenges and Opportunities with Technology, 2nd ed, available at https://www.salesforce.org/wp-content/uploads/2020/02/2nd-edition-Nonprofit-Trends-Report.pdf [iii] Callis, Z., Seivwright, A., & Flatau, P. (2019). [iv] Social Ventures Australia (n.d.) Managing to Outcomes: A Guide to Developing an Outcomes Focus, available at https://www.socialventures.com.au/assets/SVA-Outcomes-Management-Guide.pdf [v] Callis, Z., Seivwright, A., & Flatau, P. (2019). [vi] Mollenhauer, L. (2016) Board Oversight of Not-for-profit Board Evaluation, Chartered Professional Accountants Canada, available at https://www.cpacanada.ca/-/media/site/business-and-accounting-resources/docs/g10076-rg-board-oversight-of-nfp-program-evaluation-october-2016.pdf?la=en&hash=CE05AE5E06EE3F9E4AB770EA793F70482057463E [vii] Salesforce.org (n.d). [viii] Social Ventures Australia (n.d.).