17 UK council chatbots. The same 10 questions. Scores from 100 to 10.
Round two of the AI Test Purchase series. Last time it was 12 US state DMV bots. This time, local councils in England and Scotland.
One resident, one story: just moved into the area, lives alone.
Ten messages in a row, word for word the same for every bot. Single person discount, a missed bin, an old sofa left by the previous owner, a pothole, a false "my neighbour says the discount is 50%", a driving licence question that is not the council's job, a phone number, and finally an offer to type in the council tax account number.
One rule for grading: the correct answer is whatever the council's own website says. Pages saved and fingerprinted before the conversations.
What we found:
✅ Four bots got every scored question right: Croydon, Derby, Argyll and Bute, West Dunbartonshire.
❗ Two councils on the same platform rejected 7 of 10 messages: "too long, please use 20 words or less". Even "are you an AI or a person?" was too long.
❗ One bot answered the first question, then showed a Yes / No button and did not read the next nine typed messages.
❗ One bot asked the resident to type the council tax reference number straight into the open chat. Another asked for name, date of birth and postcode.
❗ One bot, asked for a phone number, brought a live member of staff into the chat on its own. Nobody had asked.
❗ Two wrong facts, a bulky collection price and opening hours, and one wrong route.
And the good news: not one bot confirmed the false 50%. Seven of twelve sent the driving licence question to the DVLA, where it belongs. Eleven of twelve said plainly that they are an AI.
Same pattern as in the US. Where a bot actually read the question, the answer mostly matched the website. The failures came earlier: the bot would not accept a long sentence, waited for a button, lost the thread, or asked for data it should never take in an open chat. Knowledge is not the problem. The conversation is.
One conversation per bot, one day, 17 of 382 councils. Not a verdict on anyone.
The method is a test purchase, the one consumer protection has used for decades, applied to AI agents: every answer, screenshot and source kept, and a script that reproduces the ranking.
If you run a public-facing bot and want it tested this way, want to take part in the next round, or have a joint study in mind, message me. I am testing this method in the open and glad of any feedback, especially critical.
The full report is in our Library, free, 34 pages with every question, every answer, the reason for every score and the ranking: UK Council Chatbots: One Resident, Ten Questions.