Given all the hype around generative AI, it was only a matter of time before someone had the bright idea of using it to dispense business advice.
After all, if ChatGPT and its clones can (or so we’re told) write original copy in the style of some of the giants of literature, surely they can manage a little friendly wisdom on cost cutting and growth strategies?
That was no doubt the thinking of the New York City Department of Small Business Services when it launched its MyCity AI chatbot – a tool designed to help residents in the Big Apple interested in launching their own business.
As the saying goes, what could go wrong?
Unfortunately for NYC administrators, the app quickly became a case study for AI naysayers everywhere to rally around when it allegedly started “hallucinating” – a technical term for AI tools making things up.
Apparently, this is a known risk with AI that operates on large language models (LLMs) – massive language-based data resources. LLMs are great for helping AI tools generate text that reads just like a person has written it, which is what has blown everyone away about generative AI. In essence, the algorithms analyse the patterns of massive volumes of writing so precisely that they become highly skilled at copying its structures.
What they’re not so good at is appreciating the line between fact and fiction. ‘Hallucinating’ is the AI equivalent of a fiction writer absorbing a load of facts and then spinning an interesting yarn from them. Which isn’t what you want when you’re looking for business advice.
So it was that the maligned MyCity bot got caught telling prospective employers that they were allowed to take their employee’s tips, even though that’s against the law. As is not accepting cash in NYC, although the app again told users they could go ahead and go cashless.
Evolving capabilities
So is all the proof we need that AI isn’t ready to take on the serious business of dispensing financial and commercial advice?
Perhaps not in its current state of sophistication. But the big caveat that needs to be added is that AI is evolving and improving all the time.
AI systems like LLMs depend fundamentally on the data they are ‘trained’ on. ChatGPT is in no small part capable of astonishing feats of human-like literacy because it has, in effect, been trained on the entire internet. But as the internet is filled with more than its fair share of dubious material, users are cautioned to be on their guard about what they take at face value.
Nonetheless, LLMs get better at pattern recognition and learning how to present ‘correct’ responses to requests the more data they have at their disposal. NYC officials suggested that the MyCity app was trained on the administration’s own self-help business resources, a smaller sample than ChatGPT’s by some orders of magnitude. It’s not surprising that it should be less reliable as a result.
A big challenge for developing training model data sets for AI financial and business advice is the compliance element. As everyone in business knows, rules and regulations abound, and they vary considerably from place to place. The minutiae is not an issue for AI – one of its strengths is filtering through dense detail at a technical level. But as the MyCity app demonstrated, understanding which rules apply, when, where and to whom, and how multiple regulations influence each other – that’s a challenge even for AI to unpick.
Ultimately, it all comes down to how the algorithms are programmed and the models trained. Developers will no doubt be working on AI business advisory apps as we speak. But in the meantime, existing tools can be used the same way as any other online search or digital tool – with a critical eye and a commitment to fact checking all you discover before you act on it.
And besides, you can still do things the old fashioned way and speak to an experienced, professional, human expert.
