AI Bias Discussion

Hi everyone. It's been a while since I've posted here, and not for lack of news, particularly news involving business decision making. Recently I wrote an article for the Abundance Institute which I'll post in full here. As a data science professional, I've been keeping up with the landscape of AI regulation, and the discussions around AI Ethics. Suffice it to say, the discussions are broadly operating from a place of fear. I do not want to minimize concerns about AI, but we have ways to observe and learn about the behavior in AI, these methods are tried and true, and we use them all the time in the sciences to evaluate opaque problems. We should be confident in our current capabilities and skeptical of additional rules and regulations on AI companies that would entrench incumbents and stymie competition. Here's my article in full:

The concept of “AI Ethics” has had a foothold in tech discourse since the mid 2010s, with particular focus on the impact of “black box” machine learning models that underpin AI. This article by Konstantine Arkoudas does an excellent job of highlighting some of the misplaced and misallocated concerns about AI, “algorithmic justice”, and bias in general. It served as a good reminder that there’s a need to return to the basics of  algorithmic bias, and the methods to test for bias, and what, if anything, is different given today’s technology. 

What is Bias in the Context of AI?

In classical statistics, bias is a straightforward measurement of the difference between algorithmically predicted outcomes and the real world outcomes we observe, or the real world outcomes we think the model should produce. We can measure and correct for this bias in some models. In other models it is better to “retune” the model to better account for the bias. Any kind of model can produce bias, whether the model of the world is some simple average of the values or outcomes we’ve seen historically, or if the model is a more complicated classical linear regression model. What’s different now though is that with our increased ability to leverage compute power, we can see, observe, and measure bias in machine learning and AI models. Importantly, as Arkoudas points out in his piece, in algorithmic modeling we know precisely what inputs were used, unlike when we evaluate human bias. With human biases we can only rely on a human to explain their decision making process.

The Three Sources of Bias in AI Models

There are three main sources of error in an AI model. Below I’ll outline each source of error and an example of how that error could come about:

• Underlying Data: If the input data supporting your model systematically under-represents certain demographic groups, or the outcomes you are concerned with are biased towards certain demographic groups in the underlying data, the resulting AI or ML model will likely exhibit some bias
  • Consider a model that would be used to evaluate mortgage loan applications for both approval or denial and for the interest rate on the loan. If that model used data from the 1950s, when black families were systematically denied loans or given usurious interest rates on loans, the resulting model would retain that bias. NOTE: it is insufficient to mask demographic data from the model, because models can use combinations of seemingly unrelated variables to “unmask” demographic data. 
• Data Processing: There are numerous steps involved in data processing that might omit data, impute data, or reclassify data. Any of these steps can change the data in a way that systematically underrepresents or misrepresents demographic groups. 
  • Consider the same lending model as above. If the lender decided to drop every non-approved application from the data, in order to focus on interest rate decisions for approved applications, all of the application data from black families who were denied loans would be removed, and white families would be over-represented in the approved loan set. 
• Algorithm: Even after the data is compiled and processed, it is possible (as discussed above) for the algorithm to “unmask” demographic information and use that information in a way that is not explicit to the model developer. This could result in a model that, even with unbiased data inputs, outputs results with a bias towards certain demographic groups. This is a more difficult problem to solve, and usually involves “retraining” the model using different hyperparameters or reweighting the underlying data.
  • Consider the same model described earlier, but with appropriate and seemingly unbiased input data. If this model is fed two applications, one from a white applicant and one from a black applicant, with identical non-demographic data fields, but the output either denies the black applicant and not the white applicant, or offers the black applicant a higher interest rate than the white applicant, the algorithm itself may be unmasking demographic characteristics and making biased outcome decisions. 

How to Fix Bias Errors

So how do we test for bias before it becomes a problem? This is straightforward in the underlying data and data processing steps. We can use normal statistical techniques to test whether demographic variables are adequately represented in our data set compared to our population. Additionally, we can use normal statistical techniques to evaluate the effect of demographic variables on key outcome variables in our data set. In our example case, we can test demographic variables on loan approval and interest rate outcomes in a more naive model. It is important to note that some data sets exhibit a known bias that should not necessarily be corrected. For example, a data set of Ferrari owners will overrepresent high income and high net worth individuals. If the model the data is supporting is only relevant to Ferrari owners, then introducing low income and low net worth individuals to the data for the sake of alleviating bias would be improper, and would result in an inaccurate model. 

Once we are confident in our underlying data, we can move on to testing the algorithm. Black box tests are the most common method to evaluate an algorithm but the design of these experiments is critical to the test. These Black Box tests are tests designed by keeping in mind that it is difficult to observe the inner-workings of a machine learning model, so we design experiments to test how the model converts inputs to outputs and repeat that test some sufficient number of times under appropriate experimental conditions. In my example of an algorithm misfiring seemingly on demographics, we used two loan applications with identical information outside of demographic information, and observed divergent outcomes. This could be due to model noise, or it could be systemic. In order to test whether bias is systemic we need to identify a clear control demographic group, and a clear test demographic group. For example, if we wanted to test whether the model outcomes are biased against black individuals, we need to decide if our control group is white individuals, or individuals from all other demographic groups. 

Once we have identified our control groups and test groups, there are two types of tests we might consider doing. The first would be adversarial testing. In this test we take a sample of data from our control group, with known outcomes, and change the demographic data to mimic our test group. Then we observe the results and see if they are statistically different from our control results. The second is a test of our confusion matrix. A confusion matrix evaluates how well a model predicts using data with known outcomes, and helps us evaluate the ratio of correct predictions compared to incorrect predictions. This is useful for evaluating overall model performance and potential model bias. In this case we take a sample from our control group, with known and/or desired model outcomes, as well as a sample from our test group with known and/or desired outcomes. Then we input those into the model and record the outputs, and then group them into a confusion matrix. This matrix shows us true positive outcome, true negative, false positive, and false negative outcomes for our control group and our test group based on our known/desired outcome evaluated against our model outcome. Then we test whether the false positive and false negative rates differ between our control group and test group. 

The most critical element of this testing is to clearly define the control and test group, and to clearly define the outcome being tested. If we set up an experiment, but start observing and measuring different outcomes from what we planned to test, we are introducing errors into the test and could draw incorrect conclusions. 

The Tools to Combat Issues of Bias Exist

AI algorithms have certainly increased the complexity of our models and in turn increased their potential scope of applications. Yet despite these advances in technology, the way we test for, identify, and evaluate bias remains the same. A model is nothing more than a way of evaluating information, determining what information is critical, and describing the world based on those determinations. As long as we have measurable inputs and outcomes we have the capability to design appropriate experiments for bias. In fact, these methods are already used to evaluate business compliance with all kinds of regulation, from employment regulations to fair lending regulations. For example, the FDIC uses statistical analysis to determine which lending institutions might be at higher risk of using discriminatory practices.

What researchers, policymakers, journalists and anyone concerned about AI bias should take away is that there is a toolkit to deal with bias issues. Developers are constantly deploying and improving these methods. These methods are sound and have a long history, and we should not feel that our testing toolkit is inadequate just because the algorithms have become more sophisticated.