Selecting the Best Data Processing & Analysis Tool

Owned by Kaley Lambden
Last updated: Apr 25, 2024 by Gillian Javetski
2 min read

No two CommCare projects are the same. As such, you may find different requirements for your data processing and visualization needs that may evolve over time. This page outlines different CommCare data processing and analysis options for you to make the best choice.

Every CommCare project must eventually interpret the Case and Form data collected by mobile workers. There are two key questions embedded in the design of a data pipeline: the method of export (eg. basic export interface) and the automation of analysis (eg. VBA queries). Both of these assumptions should be regularly revisited in projects that are planning to do large-scale data analysis (~50,000+ rows exported at a time). 

Method of Export

Export Method

 

Scale

Requirements

Export Method

 

Scale

Requirements

E1

https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143954661

0 - 50,000 rows

None

E2

https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2143951612 *

0 - 500,000 rows

None

E4

https://dimagi.atlassian.net/wiki/spaces/commcarepublic/pages/2198503499

10,000 - 500,000 rows

PowerBI/ Tableau Desktop

E3

writes to an Excel file

10,000 - 500,000 rows

  • Excel license

  • CommCare Data Export Tool set up

E5

writes to a Dashboard

50,000 - 1,000,000+ rows

  • CommCare Data Export Tool set up

  • Database installed and configured

  • Best practice: Dedicated server to run CommCare Data Export Tool

*Note: Daily Saved Exports are pre-compiled data exports. This means that when you go to CommCare HQ, you can download fresh data immediately instead of waiting for a new file to be generated.

Automation of analysis



Analysis approach

Export method

Scale

Requirements



Analysis approach

Export method

Scale

Requirements

A1

Export into Excel for manual analysis*

E1, E2, E3

0 - 200,000 rows

  • Excel license

A2

Export into Excel and use macros for analysis

E1, E2, E3

1,000 - 200,000 rows

  • Excel license

  • VBA expertise

  • Helps to have a general sense of computational complexity to avoid performance concerns

A3

Export into a CSV and use either: a scripting language (Python, Ruby, Perl, etc), stats package (Stata, SPSS, SAS, R, MATLAB, etc), or business intelligence software (Tableau, Google Fusion Tables) for analysis



E1, E2, E3

50,000 - 1,000,000+ rows

  • Analysis software or programming language installed

  • Programming background

A4

Export into a database and use database queries (SQL, etc) for analysis

E4

50,000 - 1,000,000+ rows

A5

Export into a database and use a web service to dynamically query the database

E4

50,000 - 1,000,000+ rows

  • Database installed and configured

  • CommCare Data Export Tool set up

  • Web service installed and running on servers

  • Dedicated software engineer

*Note: Depending on the complexity of indicators being calculated, this option does go beyond pivot table capabilities and not be a viable option regardless of the number of rows.