‘I’m just a number”, a common response of individuals treated badly by their employer. But the ‘bread and butter’ of AI and NHS Talking Therapies is data points. Just as an employee might protest that they are more than a unit of production, so to clients may protest about a reduction of their experiences to a score on a questionnaire they completed.
NHS Talking Therapies boasts more than a million clients a year, with questionnaires completed on each after every session. There is no doubt that they have accrued voluminous data. But the data does not speak for itself. A score on a questionnaire has no intrinsic meaning. Multiplying the number of questionnaires completed provides no added evidence of anything.
It is possible to treat the results of trained clinicians as a ‘gold standard’ against which to assess the performance of Chatbots. But the vetting procedures for the ‘gold standard’ needs to be robust. There has been no independent evidence that what trained clinicians generally do makes a ‘real-world’ difference. A consensus judgement about ‘good practice’ is not evidence. As in a Court of law, evidence must be based on multiple independent assessments and a variety of information sources. In lieu of a reliable database, appeal is made to the results of randomised controlled for depression and the anxiety disorders. But this is to ‘impersonate ‘ the rcts, the latter had independent assessments of the diagnostic status of clients before and after treatment and follow-up. Comparability of effectiveness with the rcts is a ‘con’.
It is quite possible that Chatbots do just as well or badly with sufferers from anxiety/depression as NHS Talking Therapies. But this is because of the employment of metrics of convenience, rather than validity. The irony is that this could be seen as a cost-saving way of eliminating at least low-intensity therapy. The Service has backed itself into a corner, hoisted by its’ own petard.
Dr Mike Scott