Studies have shown that discrimination exists in datasets labeled by crowdsourcing. To ensure data fairness, the authors of this paper propose a way of finding optimal task-assigning strategies within fairness, diversity and budget constraints.
Their method uses accuracy matrices to characterize crowdsourcing workers. This information is then used to distribute tasks among workers. One can estimate accuracy matrices of workers in real world by using gold tasks. Finding a crowdsourcing strategy is then an optimization problem that can be solved by linear programming.