Can we trust AI on climate information? When AI models abandon facts...
Document details

Can we trust AI on climate information? When AI models abandon facts to please users, fail at complex reasoning, and lose accuracy across languages

Contains fiugures, bibliogr. pp. 25-26

CC BY-SA

"33 large language models (LLMs) tested, 5 of them across 13 languages.
1. Policy reasoning: AI represents climate facts well, but fails on complex climate policy. Today’s leading AI models can readily explain established climate facts, but they struggle as soon as policy questions require systems thinking. Tested on a single measure, such as whether a carbon tax will reduce projected warming, the models generally answer correctly, reaching up to 94.5% accuracy. Yet when two policies are combined, accuracy collapses to 33.8%. Asked whether two combined policies achieve more, less, or the same, models scored, identical to random guessing. Even the largest frontier systems show no improvement on this task. Since meaningful climate action relies on overlapping policies working together, AI outputs cannot replace validated simulation models or expert analysis.
2. Language: Access to reliable climate information depends on the language you speak. The quality of climate information a model provides varies sharply with language. Average accuracy reaches 88.6% in English and stays within about one percentage point in Spanish, French, and Portuguese. But performance drops by 5 to 7 points in Bengali, Hindi, and Urdu, and falls by 10 points in Swahili. These lower-resource languages are spoken across South Asia and East Africa, regions identified by the IPCC as among the most vulnerable to climate change. An inquiry that receives an accurate response in English can yield inconsistent or misleading guidance in Swahili. This gap is not unavoidable: the most language-robust model lost less than 3 points in Swahili, while the weakest lost over 16. The divide is actionable: it stems from model design and training data, not the languages themselves.
3. Sycophancy: Under conversational pressure, AI routinely abandons scientific facts. 8 in 10 concessions came at the first objection. When users push back, AI systems tend to prioritize agreement over scientific accuracy. In tests across 15 models presented with settled climate claims, such as whether carbon dioxide drives global warming, starting from answers they had got right. When a simulated user challenged those answers, escalating from disagreement to false credentials and fabricated evidence, every model yielded, driving accuracy down by 31 to 35 percentage points. In roughly eight out of ten cases, models gave up the correct answer at the very first objection, before any credentials or counter-evidence were offered. Over sustained exchanges, models conceded just as quickly to sceptics as to scientists. A system that can be talked out of basic scientific facts cannot serve as a dependable guide on climate change." (Short summary)