By Kit Eaton

The complex math-based systems that buzz away at the heart of modern AI systems require huge amounts of data to train them to be useful when users ask for help. But that input data, which is obtained from many sources, obviously shapes what the chatbots “say” when you prompt them, and any biases built into the data may emerge. Knowing this, OpenAI wanted to know how fair or unbiased ChatGPT was, so it did some experiments to test one really important aspect: what impact a user’s name had on how the AI responded. Pleasingly, the chatbot scored really well. But there are still lessons and warnings for AI users in the results of OpenAI’s investigations.

In a recent company blog post, OpenAI explained that when training AIs it hones “the training process to reduce harmful outputs and improve usefulness.” Still, it notes that internal research has “shown that language models can still sometimes absorb and repeat social biases from training data, such as gender or racial stereotypes.” To probe this, the company wanted to explore how ChatGPT responded to a user based on “subtle cues about a user’s identity—like their name.” It matters, OpenAI said, because people use chatbots like ChatGPT “in a variety of ways, from helping them draft a resume to asking for entertainment tips.” Though other AI “fairness testing” has been carried out, it often relies on different more esoteric scenarios studied for bias, such as “screening resumes or credit scoring.” OpenAI is basically saying it’s aware there are subtle variations in ChatGPT, and it wanted to shine a light on them.

The task makes sense: after all, we live in an era when AI use is rising fast, but so is awareness of the risks and pitfalls of using it. As long ago as 2018 it was known that AI systems could show “unconscious” biases. In particular, AIs like ChatGPT may mirror subtle human biases based on assumptions of race or gender that manifest themselves when you hear someone’s name. “Names often carry cultural, gender, and racial associations, making them a relevant factor for investigating bias,” OpenAI noted, highlighting that “users frequently share their names with ChatGPT for tasks like drafting emails.” Because ChatGPT can now “remember information like names across conversations,” this sort of information, and possible accompanying bias, is persistent.

Luckily its experiments showed that when the same AI prompts came from users with very different user names, OpenAI’s ChatGPT chatbot showed “no difference in overall response quality for users whose names connote different genders, races or ethnicities.” And where “names occasionally do spark differences in how ChatGPT answers the same prompt,” the experiments showed less than 1 percent reflected a “harmful stereotype.” It didn’t elaborate on what a “harmful” output could be, but it may be as innocuous as the AI shaping a fairytale around a hero or a victim persona when asked to write one…or could be much more serious.

The results weren’t 100 percent flawless, however, and ChatGPT did show small biases when prompted to answer on different topics. For topics like legal matters, tech questions or health issues, less than 0.01 percent of answers showed “harmful” stereotyping in the output. But for entertainment it was much worse, with 0.234 percent of answers being harmful. Art questions and business and marketing topics also triggered more biased output than most other subject matter.

These percentages are tiny, and they may even be well below the kind of stereotyped thinking that the average human on the street exhibits.

But there’s still a critical lesson here. Even though this test concentrated on one tiny facet of using a chatbot—a user’s name—small biases in its outputs were definitely measurable. And there may be all sorts of other still undiscovered biases that emerge from an AI when you type in a query.

If your company is one of those that’s racing to use AI to help draft letters to clients, dream up marketing content, or any one of a thousand other little tasks, be careful to check its output doesn’t exhibit a bias that could accidentally cast your company in a bad light.