Decade-old CNN outperforms latest LLMs at skin cancer diagnosis

A practicing dermatologist compared the diagnostic accuracy of two frontier large language models—GPT-4o and Claude 3.5—against a convolutional neural network from a decade ago on skin cancer diagnosis tasks. The older CNN model significantly outperformed both modern LLMs on this specialized medical task, despite the widespread assumption that newer, larger models are categorically better at everything.

The finding highlights a persistent gap: frontier LLMs, trained on broad internet data, often underperform on narrow, specialized domains where older, purpose-built models were optimized. This reflects a real tension in AI deployment: bigger and newer does not always mean better for domain-specific problems, especially in high-stakes areas like medical diagnosis.