The Remote Audit: An Inexpensive Satellite-based Check on Experimental Integrity
[Paper link]
What Is a Remote Audit? In randomized controlled trials (RCTs), valid causal inferences hinge on true randomness in treatment assignment. Traditional balance checks rely on in-person baseline surveys, which can be expensive, delayed, or entirely infeasible in data-scarce contexts.
A remote audit uses pre-treatment satellite imagery and machine-learning---driven feature extraction as an early-stage, design-based diagnostic to verify that treatment was genuinely assigned at random---before any costly field data are collected.
How It Works
- Gather Pre-Treatment Images
For each experimental unit, obtain high-resolution Earth-Observation (EO) data (e.g., Landsat, Sentinel) from the period before the intervention. - Extract Image Representations
Apply vision models (e.g., CLIP, Swin Transformer, Vision Transformer) or simple indices (e.g., NDVI) to convert raw imagery into compact feature vectors. - Train a Predictive Model
Fit a classifier (logistic regression, neural net) to predict treatment assignment from these pre-treatment features. - Perform a Conditional Randomization Test
- Test statistic (T): Measures how well the model’s out-of-sample predictions exceed chance (e.g., log-likelihood gain, AUROC).
- Null distribution: repeatedly re-sample treatment assignments according to the original randomization scheme (complete, stratified, cluster) and recompute T each time.
- p-value: proportion of resampled T’s at least as large as the observed T.
- Interpret Results
- A small p-value signals that pre-treatment image features predict treatment better than chance—flagging a potential breach in randomization.
- A large p-value provides added confidence that no obvious, image-detectable biases influenced assignment.
Why Use a Remote Audit?
- Early Detection: Catch implementation errors or non-random assignment before launching field surveys.
- Global Reach: Works anywhere you have satellite coverage—perfect for low-resource or hard-to-access regions.
- Design Flexibility: Compatible with any known randomization scheme, including stratified and cluster designs.
Caveats & Limitations
- Invisible Factors: Political ties, social networks, or community leadership structures rarely leave visible traces in satellite data.
- Data Quality: Results depend on image resolution, cloud cover, and licensing restrictions.
- Audit ≠ Guarantee: Passing the audit adds confidence—but does not prove perfection. It simply shows no image-detectable biases or imbalances.
Case Study: Uganda Youth Opportunities Program (YOP)
- Context: A 2008 government RCT in rural Uganda awarded business-plan grants (~$7,500) to randomly selected youth groups.
- Remote Audit:
- Extracted pre-treatment Landsat imagery for each village.
- Trained three embedding models (CLIP, Swin, ViT) to predict grant receipt.
- Used the permutation-max procedure to control for testing across multiple models.
- Outcome: The observed test statistic fell well within the null distribution, indicating no detectable image-based imbalance---and thus reinforcing confidence in the original randomization.
See our paper for additional information.
Get Started
- Plan Ahead: Include a remote audit in your RCT pre-registration protocol!
- Choose Your Imagery: Select public (Landsat, Sentinel) or commercial (PlanetScope, WorldView) based on your resource constraints and spatial/temporal needs.
- Select Embedding Methods: Balance interpretability (e.g., NDVI) against predictive power (e.g., foundation-model embeddings).
- Run the Audit: Use our open-source audit toolkit to automate resampling and test-statistic computation.
- Interpret & Report: Share audit p-values alongside traditional balance tables to strengthen your study’s credibility.