their most powerful model, to label all of the neurons in a much smaller model, GPT2. The researchers were able to find multiple neurons that seemed to correspond to recognizable concepts, such as a neuron that seemed to activate for ‘doing things right’.
In practice, explainability would need to be implemented differently depending on which AI system is being used and what it’s being used for. It would be left to the AI developers to work out how to do this in each case. The difficulty arises from AI developers’ inability to provide any reliable explanation for decisions made by more complex systems, such as LLMs. If legislation was passed that required all AI systems to be explainable regardless of their architecture, it could spell disaster for companies developing these more complex systems.
In the second scenario, regulators, aware that explainability is not currently feasible for more complex AI systems, could create carve-outs for lower risk tasks or only partially enforce the legislation that Congress enacts.