Assessors cannot override wrong decisions

A Commonwealth Ombudsman investigation has concluded that the aged care algorithm used to decide how much support older Australians receive is unreasonable, unfair and lacks human oversight. The finding comes in a 104-page report on the Integrated Assessment Tool, led by Ombudsman Iain Anderson. Under the law, the tool's automated funding recommendations cannot be overridden and changed even when they are clinically inappropriate, the report states.

Anderson said the legislation must be changed so assessors can themselves alter clinically incorrect algorithmic decisions.

The problem was visible internally before the tool went live. Senior staff warned in one email that the review pathways only allow for the algorithm outcomes to be confirmed, and that the framework did not appear to give people affected a clear path to seek review.

A technological black box

The report is also scathing about transparency. It says there is no public information available that meaningfully explains how the algorithm converts inputs into outcomes, and that without this the Integrated Assessment Tool is a technological black box.

On fairness, the report went further than flagging a risk. It found that a reasonable person would lack confidence that the assessment system is in fact fair and consistent. Anderson added that the inputs provided by assessment providers are really critical to what the algorithm produces, meaning the department must oversee those providers far more effectively.

Rollout problems and missed consultation

The Integrated Assessment Tool changes were rolled out in November, and people involved in developing it had no idea the government would add an algorithm. Freedom of information documents show that officials realised only in the hours before the rollout that legislation legally barred human override, followed by departmental panic, severe clinical distress among frontline assessors and secret IT workarounds within days.

• •Claims of years of trials and consultation supporting the tool were found to be incorrect
• •Those trials and consultation related to earlier versions of the IAT and earlier legislation
• •There was no meaningful consultation on the version that went live

Anderson said it was very concerning that the department was telling people the system would operate in a particular way and then changed it. The department also failed to enforce contractual quality controls or verify whether assessors held mandatory clinical qualifications, despite assessor skills being critical to the tool, according to the report.

Robodebt lessons unheeded and next steps

The investigation, triggered in April by concerns about transparency of how the algorithm makes decisions for support at home packages, found the department has not complied with recommendations of the royal commission into the robodebt scheme regarding transparency of government automated decision-making. Anderson said when government uses automated decision making or artificial intelligence, it needs to be very transparent so people understand decisions that affect them significantly.

In July the Department of Health and Ageing announced a new review process allowing assessors to escalate wrong decisions to the department, but Anderson said he is concerned the escalation pathway will not necessarily fix the problem.

The investigation also identified numerous legal concerns awaiting government address. "How can you actually challenge decisions when you don't know how they're made by the algorithm?" Anderson asked. He said the significant legal issues need addressing urgently, and that the department did not, and may still not, understand key components of the legal framework it created. The report calls for a comprehensive review of the legislation related to the needs assessment process and of the rules underpinning the algorithm.