Endowing imperfect ML models with agency in a crowdsourcing platform increases analytic throughput while maintaining data quality.