This page is a list of background readings. It includes:
- Accessible Introductions to the Problem
- Research Framings and Overviews
- Algorithm Audits by Researchers
- More General Algorithm-Related Sources
- Audit Methods in Other Domains
Help us improve this list! Please submit suggestions for additional resources to “algo.audits@gmail.com”
Accessible Introductions to the Problem
Sandvig, C., Hamilton, K., Karahalios, K., & Langbort, C. (2014). “An Algorithm Audit.” In: Seeta Peña Gangadharan (ed.), Data and Discrimination: Collected Essays, pp. 6-10. Washington, DC: New America Foundation.
Hodson, H. (2015, February 4). “No One in Control: The Algorithms that Run Our Lives.” New Scientist 3007. http://www.newscientist.com/article/mg22530070.300-no-one-in-control-the-algorithms-that-run-our-lives.html (subscription required).
Angwin, J. (2014). “Hacked,” In: J. Angwin, Dragnet Nation: A Quest for Privacy, Security, and Freedom in a World of Relentless Surveillance, pp. 1-20. New York: Henry Holt & Co. http://juliaangwin.com/wp-content/uploads/2014/02/DragnetNation-Chapter1.pdf (Ch. 1 available free online.)
Walker, B. (2014, October 6). “Enchanting by Numbers.” Theory of Everything. Cambridge, MA: Public Radio Exchange. http://toe.prx.org/2014/10/enchanted-by-numbers/ (audio podcast: 24.56)
Pasquale, F. (2015). The Black Box Society: The Secret Algorithms that Control Money and Information. Cambridge, MA: Harvard University Press. http://www.hup.harvard.edu/catalog.php?isbn=9780674368279
Research Framings and Overviews
Sandvig, Christian, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. “Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms.” In Data and Discrimination: Converting Critical Concerns into Productive: A preconference at the 64th Annual Meeting of the International Communication Association. Seattle, WA, 2014. (This paper inspired the present workshop. –Ed.)
Diakopoulos, N., “Algorithmic Accountability Reporting: On the Investigation of Black Boxes” (accessed March 2, 2014).
Hamilton, Kevin, Christian Sandvig, Karrie Karahalios and Motahhare Eslami. “A Path to Understanding the Effects of Algorithm Awareness.” In CHI 2014. Toronto, ON, 2014.
Barocas, S., Hood, S., & Ziewitz M. (2013). Governing Algorithms: A Provocation Piece. Available at SSRN: http://dx.doi.org/10.2139/ssrn.2245322
Algorithm Audits by Researchers
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K., Hamilton, K., and Sandvig, C. (2015). “I always assumed that I wasn’t really that close to [her]:” Reasoning about invisible algorithms in the news feed. Proceedings of the 33rd Annual SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery (ACM)
Eslami, M., Aleyasen, A., Karahalios, K., Hamilton, K., and Sandvig, C. (2015). FeedVis: A Path for Exploring News Feed Curation Algorithms. Software demo presented to the 18th Annual Association for Computing Machinery (ACM) Conference on Computer-Supported Cooperative Work (CSCW).
Hannak, A., Soeller, G., Lazer, D., Mislove, A., Wilson, C. (2014). Measuring Price Discrimination and Steering on E-commerce Web Sites. ACM SIGCOMM/SIGMETRICS Internet Measurement Conference (IMC ’14).
Hannak, A., Sapiezynski, P., Kakhki, A. M., Krishnamurthy, B., Lazer, D., Mislove, A., Wilson, C. (2013). Measuring Personalization of Web Search. Proceedings of the 22nd Int’l World Wide Web Conference (WWW ’13).
Mathias Lecuyer, Guillaume Ducoffe, Francis Lan, Andrei Papancea, Theofilos Petsios, Riley Spahn, Augustin Chaintreau, and Roxana Geambasu. “XRay: Increasing the Web’s Transparency with Differential Correlation.” Technical report, July 2014.
Edelman, B. (2011). Bias in Search Results? Diagnosis and Response. Indian Journal of Law and Technology 7: 16-32.
Edelman, B. & Luca, M. (2014). Digital Discrimination: The Case of Airbnb.com. Working paper.
More General Algorithm-Related Sources
Anderson, C.W. (2012). “Towards a Sociology of Computational and Algorithmic Journalism.” New Media & Society.
Baker, P. and A. Potts. (2013). “‘Why Do White People Have Thin Lips?’ Google and the Perpetuation of Stereotypes Via Auto-Complete Search Forms.” Critical Discourse Studies 10, no. 2: 187- 204.
Beam, M.A. (2013). “Automating the News: How Personalized News Recommender System Design Choices Impact News Reception.” Communication Research.
Benjamin, S.M. (2013). “Algorithms and Speech.” University of Pennsylvania Law Review 161, no. 6: 1445-1494.
boyd, d. and K. Crawford. “Critical Questions for Big Data.” Information, Communication & Society 15, no. 5: 662-679.
Brunton, F. and H. Nissenbaum. (2011). “Vernacular Resistance to Data Collection and Analysis: A Political Theory of Obfuscation.” First Monday 16, no. 5.
Calo, R. (2011). “Peeping Hals.” Artificial Intelligence 175, no. 5-6 (2011): 940-941.
Clerwall, C. (2014). “Enter the Robot Journalist: Users’ Perceptions of Automated Content.” Journalism Practice.
Crawford, K., “The Hidden Biases in Big Data” http://blogs.hbr.org/cs/2013/04/the_hidden_biases_in_big_data.html (accessed August 27, 2013).
Fuller, M. and A. Goffey (2012). “Algorithms.” In Evil Media, 69-82. Cambridge, MA: MIT Press.
Galloway, Alexander R. (2004). Protocol: How Control Exists after Decentralization. Cambridge, MA: MIT Press.
Gillespie, T. (2014). “The Relevance of Algorithms.” In Media Technologies: Essays on Communication, Materiality, and Society, edited by T. Gillespie, P. Boczkowski and K.A. Foot, 167-194. Cambridge, MA: MIT Press.
Golumbia, D. (2009). The Cultural Logic of Computation. Cambridge, MA: Harvard University Press.
Granka, L. (2010). “The Politics of Search: A Decade Retrospective.” The Information Society 26, no. 5: 364-374.
Hallinan, Blake and Ted Striphas. (2014). “Recommended for You: The Netflix Prize and the Production of Algorithmic Culture.” New Media & Society.
Hazan, J. G. (2013). “Stop Being Evil: A Proposal for Unbiased Google Search.” Michigan Law Review 111, no. 5: 789-820.
Introna, L. and H. Nissenbaum. (2000). “Shaping the Web: Why the Politics of Search Engines Matters.” The Information Society 16, no. 3: 1-17.
Jiang, Min. (2013). “The Business and Politics of Search Engines: A Comparative Study of Baidu and Google’s Search Results of Internet Events in China ” New Media & Society.
Laidlaw, E. B. (2008). “Private Power, Public Interest: An Examination of Search Engine Accountability.” International Journal of Law & Information Technology 17, no. 1: 113-145.
Mager, A. (2012). “Algorithmic Ideology.” Information, Communication & Society 15, no. 5: 769-787.
Manovich, Lev. (2000). “Database as a Genre of New Media.” AI & Society 14, no. 176-183.
Manovich, Lev, “The Algorithms of Our Lives” http://chronicle.com/article/The-Algorithms-of-Our-Lives- /143557/ (accessed August 20, 2014).
Napoli, Philip M. (2014). “Automated Media: An Institutional Theory Perspective on Algorithmic Media Production and Consumption.” Communication Theory 24, no. 3.
Pasquale, F. (2011). “Restoring Transparency to Automated Authority.” Journal on Telecommunications and High Technology Law 9, no. 235: 235-254.
Sandvig, C. (2015). Seeing the Sort: The Aesthetic and Industrial Defense of “The Algorithm.” Media-N 11(1). http://median.newmediacaucus.org/art-infrastructures-information/seeing-the-sort-the-aesthetic-and-industrial-defense-of-the-algorithm/
Tufekci, Zeynep. (2014). “Engineering the Public: Big Data, Surveillance and Computational Politics.” First Monday 19, no. 7.
Audit Methods in Other Domains
Saltman, J. (1975). Implementing Local Housing Laws Through Social Action. Journal of Applied Behavioral Science, 11(1): 39-61.
Ayres, I. & Siegelman, P. (1995). Race and Gender Discrimination in Bargaining for a New Car. American Economic Review 85(3): 304-321.
Schulman KA, Berlin JA, Harless W, Kerner JF, Sistrunk S, et al. The effect of race and sex on physicians’ recommendations for cardiac catheterization. N. Engl. J. Med. 1999;340(8):618–626.
Ridley S, Bayton JA, Outtz JH. Taxi Service in the District of Columbia: Is It Influenced by Patrons’ Race and Destination? Washington, DC: Washington Lawyers’ Comm. Civil Rights Law. Mimeo; 1989. (Can’t find. Anyone have a copy? –Ed.)
Wissoker D, Zimmerman W, Galster G. Testing for Discrimination in Home Insurance. Washington, DC: Urban Inst. Press; 1998.
Pager, D. (2007). The Use of Field Experiments for Studies of Employment Discrimination: Contributions, Critiques, and Directions for the Future. The Annals of the American Academy of Political and Social Science, 609(1): 104-33.
Pager, D. (2009). Field Experiments for Studies of Discrimination. In: E. Hargittai (ed.) Research Confidential: Solutions to Problems Most Social Scientists Pretend They Never Have, pp. 38-60. Ann Arbor, MI: University of Michigan Press.
National Research Council Panel on Measuring Racial Discrimination, The. (2004). Measuring Racial Discrimination. Washington, DC: National Academies Press. (an excellent overview of audit methodology appears from p. 103 on. –Ed.)
Auditing Algorithms From the Outside: 