It can be challenging for people to believe AI can write as well as humans, even when they see examples of this in practice. In fact, convincing our customers that AI can write a first draft as good, and far faster, than their staff (saving them massive amounts of time) is the key objection we have to overcome in our sales process.
We’ve also encountered many folks who believe that they could easily tell if a particular piece of content was written by an AI or a human, judging from the voice and language used.
However in practice, researchers have found this isn’t the case – non-experts are unable to tell the difference between AI and human-produced content.
In the research paper ‘All That’s ‘Human’ Is Not Gold’, the researchers found, without training, non-experts were only likely to get ‘Human or AI’ right 50% of the time – the level of random chance. In fact across the three types of content evaluated, news articles, stories and recipes, evaluators only correctly picked human or AI 48% of the time for news articles (sorry journalists!)
Even with training, evaluators struggled to pick the writer. The researchers tried three methods to train human evaluators but only managed to raise their accuracy to 55%, barely more than randomly drawing content out of a hat.
Many evaluators underestimated the capability of AI to write human-sounding content, or its ability to inject personality, empathy or deductive reasoning into writing.
However, evaluators were often very certain of their conclusions, justifying their selection of ‘human or AI’ based on spelling, grammar and the style or tone to make their (erroneous) decision.
Ironically the reasons that evaluators gave for their choice (Human or AI) were often contradictory, with the formality of the text, spelling and grammar errors and clarity all cited to justify both human and AI judgments.
This was also reflected by low agreement scores between evaluators, where they couldn’t form a unified opinion of whether a particular piece of content was written by a human or an AI.
AI has already reached a human-level capability to write and inject voice and personality into content, so these ways of telling content apart simply don’t work.
So the next time you read a piece of content, no matter how good or bad it may be – ask yourself whether you are sure it was written by either a human or AI (or both together).
And does that really matter?
By the way, this post was written by a human, mostly. (But you’ll have to trust us on that!)