Webinar: Learn how to build a Global M&E System

Q: What might KPIs for multi sector projects look like?

A: Mala: At GOAL our KPIs are sector specific with different indicators for different sectors to be able to report on the areas of focus from our strategy.  However, depending on what your programme does, you can have generic indicators which are not sector specific, eg. percentage of people that consider their needs are met, or state the multiple sectors which are relevant, eg. percentage of communities who report ‘good’ or ‘high’ quality of health, nutrition, and WASH services

Q: How do the new global indicators and project indicators relate to (existing) case properties?

A: We are currently attempting to use case properties to be able to compare data across apps/geographies - i.e. using the same case property name across apps so we can aggregate/compare data across apps/geographies.

Monica: At Childfund I have created some hidden values that I have saved as case properties so that when we pull up a child record we can readily see if that particular child has met that indicator.  For example, we want to track school attendance for children who are of compulsory age.  In our survey we have created a hidden value that takes compulsory age + school attendance and gives you a “yes” value.  That hidden value is then saved as a case property.  We plan to use hidden values for the program indicators so that we can track these at the individual level and so that when we pull case extracts we don’t have to pull all the related questions but only those fields that we need.  Hope this explanation helps. 

Mala: In SCOREcard, case properties are critical for use in validation and display conditions as well as lots of hidden values.  They also are helpful when it comes to setting up data exports.  I typically always add all questions as case properties in the apps I design/build.  In terms of having shared case properties across apps, it is more important when designing the apps, to remember to include similar questions eg. set demographic info, to collect the same data that are comparable and can be used in aggregation.  Whether you then set them as case properties or not, does not affect the comparability of data.

Q: Are these dashboards that we are seeing on the slide from inside Commcare or integrated with PowerBI or Tableau?

A: These are dashboards that are created in PowerBI. Both GOAL and Childfund use CommCare data feeds to directly pull in data from CommCare to Power BI. Then they create the dashboard visualizations within PowerBI.

Q: What software/applications were used to generate the dashboards and how much manual intervention was necessary to generate them regularly?

A: For both of these examples, PowerBI was the analytics tool used to generate these dashboards. If you use CommCare’s PowerBI integration, it’s a one-time set-up to connect an Odata feed to your dashboard and then create the graphs. Once you set up the initial visualizations, you can open up the PowerBI dashboard and it will automatically pull down the latest data from CommCare.

Q: For those who have spent a lot of time thinking about global M&E systems, does anyone have any insights about value-add indicators?

A: It's a lot of what INGOs and other organizations focus their work on (value-add, capacity building, etc.), but very few orgs have a way of measuring this on a global level. Interested to hear if this is information any of the orgs on this call collect.

Monica: Great question. At CF we are working on different aspects of MEL, including capacity building. We have devised an entire curriculum that we are currently delivering virtually, because of the pandemic. We are tracking uptake in terms of attendance and have some short evaluations that we are doing to see whether folks feel the material is valuable. During our last session, we had one of our country office MEL specialist co-facilitate which was great and for us validation that this was of value. We don’t track this in CommCare currently or have any rigorous m&e around it at this moment. I hear about the value add from folks at different levels of the organization but it would be nice if I could quantify this to be able to speak to our evolution and all of the work that we are doing.

Mala: At GOAL our current strategy results framework does not have specific ‘value-added’ indicators because we typically respond to crises where lots of INGOs and multilaterals work together in emergencies. It is not important for us, who does what, but more that we save lives through our combined efforts. For capacity strengthening, our KPIs are typically at the outcome level so it becomes the ‘so what’ of the capacity strengthening activities. Eg. if we work with small business owners and strengthen their financial knowledge and business skills, our indicator would be the percentage of people who have increased the profit of their SME.

Q: I would really like to know how data cleaning is managed in the integration from CommCare to Power BI.

A: Dev: We fundamentally believe that the best place to prevent bad data is at its source. Accordingly, we have built quite a few capabilities/controls on the app building side of things to address this:

#1: Strong Data Validations: You can build pretty sophisticated data validations on the app using our basic and advanced validation conditions. The mobile user can't progress ahead if a given validation condition is not met. 

#2: Auto calculations: Based on prior answers and/or case information, you can have the app dynamically auto-calculate certain values, which de-risks errors due to manual work. 

#3: Job-to-be-done optimized data entry: CommCare apps can be built in a way such that a single app can be dynamically optimized based on who the user is (user data), who the beneficiary is (case data) and other attributes like date/time. Additionally, with Case Management the case/beneficiary data can be selectively stored locally on the mobile device, the user doesn't need to enter the same information during every interaction with the beneficiary. These capabilities allow you to build user experiences that are super optimized to the job-at-hand to minimize data entry and thus resulting errors. 

For errors that just can't be prevented on the app side of things, we recommend establishing a strong feedback loop to catch such errors using our reports and/or our data auditing feature called Case Data Explorer (CDE) which Mala mentioned. After catching such errors, we recommend evaluating what can be done to prevent them from happening, which oftentimes could be minor changes to the app, process changes or end-user training. App Release Management is one of our platform's strengths, which makes deploying app changes to the field very straightforward.  

For cleaning bad data, we support in-product data cleaning capabilities as well.

Q: Can we integrate this data metabase? Or our platform through sql?

A: You can either use CommCare’s direct integration with PowerBI or you can use the CommCare Data Export tool to ship your CommCare data into a SQL database, and then feed the information in your SQL database into PowerBI or the analytics tool of your choice.

Q: Hi, it's mandatory to Subscribe to a professional plan before having advantage to use CommCare and make integration with PowerBi et Tableau?

A: Yes, this feature is available on our Advanced plan, or as an add-on to our Pro plan.

Q: For both panelists, if there's time at the end, I'm curious to hear more about your change management approach and challenges. In what elements did you get the most resistance/pushback? How did you get everyone aligned and on board?

A: Monica: Awesome question. At CF we had to convince folks that moving to mobile data collection was actually possible. What ended up happening was that one of our regional offices was the pioneer and we took advantage of their efforts to make a case for why this was the right direction to go globally. HQ is slow to move but country offices and partners have a lot more flexibility so learning from them is essential and obviously highlighting best practices. I feel that our country offices and LPs were ready for a tool so the uptake of CommCare was not difficult. We are now engaged in a process of building a global platform and have to standardize how data are collected at the field level, so we are likely to have some challenges there. I think having honest conversations along the way is important. Engaging stakeholders and elevating best practices from the field is also important, incorporating them into the design. Listening and keeping those communication channels open is very important.

Mala: This is something we have discussed a lot with the roll out of SCORE. It is a big change to move from quarterly spreadsheet reporting to an app that should be used on a weekly basis. It is also a big change for MEAL staff to now have to consolidate all results so extra time is required to do this. There were certainly no push backs at GOAL because everyone could see the benefits of the system and how it can help their work and make their lives easier. This was done through lots of induction sessions and demos with individual departments so that specific functionality relevant to the staff and their roles were presented, as opposed to just having generic talks about the new system. However, it has been difficult for some teams to find the time for the additional data entry. GOAL is project funded and being first responders to emergencies around the world means that we already work at lightning speed with few resources for day to day operations. But the MEAL staff do know that it is the initial set up and familiarisation with the system that takes time. Once it is part of everyday work with all necessary data backdated, there is much less time required to enter data whenever indicators are measured.

Q: From a customer success perspective, how much time/effort does it take Dimagi to guide a partner in building out a global M&E system?

A: That very much depends on the scope of the organization’s projects, and how similar/ dissimilar they are to begin with. A lot of the time and effort that goes into this will be done by you (the organization and project members, themselves). I would guess that you could probably do a Deep Dive Plus onboarding with our Customer Success team that would get you close to where you need to be. Just know that your internal staff might need to budget a considerable amount of their time to work on standardizing and mapping indicators :) For reference, GOAL hired an external consultant to build their SCORE reporting system in CommCare. The design was done by GOAL and then the consultant worked closely with the designer for approximately one month full-time to build the app. This included one major revision to the structure of the app, as well as testing, and setting up all the exports.

Q: Is this data being entered by M&E focal points in each country? Or is there 'front line' data collection (i.e. community level staff entering real time data to populate the info for each indicator)?

A: Monica: At CF, we rely on our local partner organizations to manage the data collection process.  They have a group of community volunteers or community mobilizers that participate in this data collection process.  They now enter data using tablets.  I mentioned that we have a core survey that is administered to every enrolled child and that we have other surveys (that are used to calculate program indicators) that are administered to a sample of enrolled children.  Some of our country offices recruit special community mobilizers or data collectors for those focused surveys.  Primarily because the questions are more complex and because some entail administering reading and numeracy assessments.  But all of these data are captured by mobile workers in the field.  

Mala: For SCORE, it is our M&E focal points entering information because the results come from a variety of other datasets and data sources which may or may not be digitized in CommCare but at GOAL we do have CommCare apps where data are collected in the field by staff, volunteers, and community members, and have calculated indicator results from them. 

Q: Does the GOAL system actually facilitate complete indicator tracking at the program level, or do programs still need to manage IPTTs in a parallel system?

A: SCORE records all indicators. The way it was built allows the user to define any indicator and track its results. It can also disaggregate indicator results in any way you tell it to with infinite possibilities and combinations. It is flexible to allow any reporting timeframe, any type of measurement, any type of entity, etc. All the indicator results can be used for strategic or programmatic purposes through the single system, removing the need for parallel or grant specific reporting tools. Except when donors still request a spreadsheet submission of results in a logframe format but these are now accompanied by exports from the dashboard as well.

Q: Are these models on sale? Are they configurable or customizable for a project and organization?

A: Dev: No. These models and dashboards were created by ChildFund and GOAL themselves using PowerBi.

Q: How do you manage data quality? To what extent can you use technical features to make sure that your dashboards are based on quality data?

A: Monica: As mentioned in my response, at CF we use all the tools available in CommCare to regularly monitor data collection.  I am a big proponent of worker reports and someone that checks data every day.  I recommend to country office MEL specialists that they engage in this practice during data collection periods.  This is a work in progress because it is not a practice that is familiar to everybody.  We also use the powerbi integration tool to create dynamic reports that make viewing data as it comes in easier. 

I have also built the application myself, therefore I have used display logic and validation logic to ensure that data will come in as clean as possible.  This makes me highly confident in my data.  I also look for outliers in my reports.  We also have some processes in place for our basic accountability where we have local partners check data on a sample to make sure data are correct and reliable.  But my feeling is that we can have greater confidence by building good apps and using the tools currently in CommCare.  

For SCORE there are many hidden calculations and validation conditions used within the app to restrict a user from entering dirty data.  However there are some cases when information entered does not look quite right.  This could be human error regarding the calculation of results or incorrectly defining an indicator, and is just improved through support and capacity strengthening.  But data quality checks should be made on a regular basis, and more frequently when a new system is deployed.  From close inspection of initial data entered into the system I knew where there were the most errors occurring so focused on finding cases with those attributed, through the case list explorer and either cleaning the data form submission directly or notifying the country office about the issue for them to clean the data. Finally, it is important to give the user easy ways to check the data they have entered and SCORE allows user edits through the app.  We also made sure that the dashboard was rolled out at the same time and not afterwards so users could see how their information is being used and see for themselves if results weren’t correct.

Q: Are you disaggregating data by gender, age, disability, etc., and how is CommCare facilitating this?

A: Monica: At CF, I make sure CommCare has all the data I need to be able to disaggregate and create the kinds of reports that  folks at ChildFund or local partners will need to make informed decisions.  So in my opinion all of this starts with a good design.  Then, it's about how to use case properties so that when you download form data for instance, you are able to bring in gender, age and anything else you may need for your report.  I try to visualize the report as I am building the application.  You can also make sure you run tests before you deploy an app.  You can do this with dummy data.  Use the powerbi integration tool and your dummy data to create mock reports.  That allows you to identify things that are missing that you need to add to your application.  In the real world we often don't have the time to do this.  But when I notice something that I am missing, I make sure I make the changes so that the next time I am collecting data I will not have to do any backend process.  

Mala: In SCORE, any indicator can be disaggregated in any way defined by the user and as outlined in their programmatic logframe.  The way the app was built, allows this free definition as there is an infinite way in which data could be disaggregated.  GOAL’s KPIs have set disaggregation where relevant, and our people reached counts are disaggregated by sex and age.  For people reached we just ask for the disaggregated numbers which are then automatically summed through hidden calculations to get a total so the user does not need to do their own calculation.  However, for indicators we ask for the total first and then the disaggregated results.   There is then validation on the disaggregated results to make sure the sum equals the total which serves to ensure clean data are entered.    

Q: Does CommCare Have the ability to collect qualitative data and the ability to do data triangulation?

A: Dev: CommCare is commonly used to collect both qualitative and quantitative data. We also support collecting data in multitude of ways-text, audio, video and more.