At Jewish Family & Children’s Service (Boston), we serve over 17,000 clients at year across 42 programs in four divisions: parents and children, seniors, people with disabilities and mental illness, and basic needs. Our agency has a long history—150 years—of serving people in the Greater Boston area. Considering the agency’s age, our four-year-old Department of Evaluation and Learning (DEL) is in its infancy, born from a vision by our Board and senior leadership to become a learning organization, while at the same time recognizing the changing fundraising landscape and public perceptions regarding holding nonprofits accountable for their outcomes.
Because most of our programs were collecting data to some extent prior to the creation of our department, one of our first tasks as an evaluation team was to address the pervasive challenge of incomplete and inaccurate data for use in evaluation. What do I mean by this? A few examples:
- Missing demographic data, for example, on client ethnicity. This is, of course, a potentially sensitive piece of data and clients should have the opportunity to opt out of answering, but it is also very important that agencies are aware of the ethnic composition of the clients they are serving. However, from an evaluation perspective, when we saw the incomplete fields, we didn’t know if items were missing because a client did respond or because those fields had been skipped during the data entry process. (Note: to address this issue, we added a “client did not respond” option to all demographic data fields—this way we know that if data are missing, it is a data collection issue, not a client response issue.)
- Lack of consensus as to what certain fields meant. For example, there was a lot of confusion on what “program end date” meant. Does it mean the date of the last contact with the client? The date that a staff member felt they had completed the work with the client (e.g., the date they completed the “closing paperwork”)? The date that the data entry was being done? (Note: to address this issue, we try to have a “codebook” for each program, with a description of what is expected in each field, to help everyone be on the same page.)
- The data gathered often did not reflect what was actually true about clients or services rendered—for example, a report indicated we had a client receiving services who was over 100 years old; perhaps not inconceivable for some of our senior services, but for a program that was intended to serve new parents, most likely a data entry error.
- Client post-service surveys that had very low response rates—if only a small number of clients respond, the data gathered is most likely biased to people who either loved the service or had complaints, which certainly doesn’t provide an accurate picture of the program.
Despite these challenges, our goal was for staff at all levels—from direct-service staff to senior leadership—to be able to make statements using data that they felt confident in and could stand behind. In addition to ensuring data integrity, there is a practical side—why even spend the time collecting a little bit of bad data if it can’t (shouldn’t) be used?
Wouldn’t it be better to take the time to focus on collecting a fewer number of fields—at least to start—with a high level of accuracy and completeness? (For a thought-provoking perspective on this topic, I would recommend reading a post on Markets for Good by Laura Quinn entitled “Forcing Nonprofits to Lie about their Data.”)
What do we mean by high quality data?
My colleague Noah Schectman and I presented at the 2014 American Evaluation Association’s annual conference as well as at the local Crittenon Women’s Union Outcomes Workgroup on this very issue—how to achieve high-quality data in nonprofit settings. We translated the “academic” ideas of reliability and validity, to be more pragmatic—at JF&CS we focus on the following four concepts when talking about high-quality data:
- Complete: the data are complete (Example 1 above).
- Uniform: data are being entered consistently across those who are doing the data entry (Example 2 above).
- Accurate: the data reflects what is true about the client or service being provided (Examples 3 and 4 above).
- Timely: the data are being entered in a manner that they can be used in real-time.
In addition, at JF&CS we use an overarching evaluation framework that Rachel Albert, Vice President of Evaluation and Learning for the agency, and I created called TIERS (“Tool for Intra-agency Evaluation Resource Sharing”) that indicates the expectations for the amount and complexity of the data being collected for each program. We created this tool as a way to respond to the important and practical question of “What data should each program be collecting, given the questions it needs to answer, for its particular combination of stakeholders and the resources available?” Together, our TIERS framework and our expectations for high-quality data allow us to create tools and processes for increasing data integrity.
Striving for high-quality data
How do we ensure these concepts—completeness, uniformity, accuracy, and timeliness—are actually implemented across the agency, however, especially considering our size? In an upcoming blog post I will be sharing the practical mechanisms that we have implemented at JF&CS in order to achieve high-quality data.
Our deepest appreciation goes out to Laura for sharing her insight and research. We look forward to learning more about her work! Feel free to share your comments with us and please share this on Facebook and Twitter using #dataintegrity. We look forward to hearing your ideas!
All images used with permission from Laura Beals, Ph.D.