Simply because aspects of artificial intelligence systems are difficult or impossible to define. … [+]
Recent years have seen a growing concern among policymakers and the public about the “explainability” of artificial intelligence systems. As AI becomes more advanced and applied to domains such as healthcare, hiring and criminal justice, some are calling for these systems to be more transparent and interpretable. The fear is that the “black box” nature of modern machine learning models makes them unaccountable and potentially dangerous.
While the desire to explain AI is understandable, its importance is often overstated. The term itself is ill-defined—what criteria make a system perfectly definable is unclear. More importantly, a lack of explanatory power does not necessarily make an AI system unreliable or insecure.
It’s true that even the creators of the most advanced deep learning models cannot fully explain how these models transform inputs into outputs. The intricacies of a neural network trained on millions of examples are too complex for the human mind to fully comprehend. But the same can be said of countless other technologies we use every day.
We do not fully understand the quantum mechanical interactions underlying chemical manufacturing processes or semiconductor fabrication. And yet that doesn’t stop us from taking advantage of pharmaceuticals and microchips that are developed using this partial knowledge. What we care about is that the outputs succeed in serving their purpose and are reliable.
When it comes to high-stakes AI systems, we should first focus on validating their performance and testing them to ensure that they behave as intended. Examining a criminal sentencing algorithm to understand how it combines hundreds of features is less important than assessing its empirical validity in predicting recidivism rates among ex-prisoners. go
An emerging field called AI interpretability aims to somewhat open the black box of deep learning. Research in this area has yielded techniques for identifying which input features are most salient in determining the model’s predictions, and for explaining how the information is used by artificial neural networks. Go through the layers. Over time, we will get a clearer picture of how these models process the data to arrive at the output.
However, we should not expect AI systems to be fully explainable in the way that a simple equation or decision tree can be. The most powerful models will likely always involve some level of intractable complexity. And that’s okay. Much of human knowledge is clear and difficult to verbalize—a chess grandmaster cannot fully explain his strategic intuition, and a skilled painter cannot fully describe the source of his inspiration. What matters is that the end results of their efforts are valuable to themselves and others.
Indeed, we must be careful not to let the capacity for explanation come to the detriment of other priorities. An AI that can be easily interpreted by a human is not necessarily more robust or reliable than a black box model. There may even be a trade-off between efficiency and explainability. Michael Jordan may not be able to explain the intricate details of how his muscles, nerves and bones coordinated to execute a slam dunk from the free throw line. Yet he managed to accomplish this remarkable feat regardless.
Ultimately, an AI system should be evaluated based on its real-world impact. A hiring model that is vague but more accurate at predicting employee performance is better than a transparent rule-based model that recommends lazy workers. A tumor detection algorithm that can’t be explained but catches cancer more reliably than doctors can. We should try to make AI systems interpretable where possible, but not at the expense of their benefits.
Of course, that doesn’t mean AI should be unaccountable. Developers should extensively test AI systems, validate their real-world performance, and try to align them with human values, especially before unleashing them on the wider world. But we must not let abstract notions of descriptive competence become a distraction, let alone a barrier, to realizing the enormous potential of artificial intelligence to improve our lives.
With the proper precautions, the black box model can also be a powerful tool for improvement. In the end, it is the output that matters, not the process that can be explained that provided the output.