Artificial intelligence could help pathologists identify breast cancer in digitized tissue samples, but understanding why a model makes a particular prediction remains a challenge, an AI researcher told the audience at Sour Mouse on Oct. 6.
Akash Parvatikar, director of artificial intelligence at BioBase, spoke to researchers, students, and technology enthusiasts about digital pathology as part of PhDs & Pints, a New York City-based series that brings academic speakers to bars across the city.
AI can process huge amounts of information without tiring and count cells faster and more consistently than humans, Parvatikar said. But its decision-making process can be difficult to trace. “It’s still a black box,” he said during the event. “It doesn’t reason the way an expert would.”
When doctors use digital imaging to study diseases, they convert traditional samples into high-resolution digital files that contain billions of pixels. AI systems can take those images, divide them into smaller sections, and analyze them faster, potentially helping pathologists review materials more efficiently.
But a system that can’t explain itself is hard to check, Parvatikar said. If an AI could “open up and show its work,” pathologists could evaluate its conclusions instead of taking them on faith.
Parvatikar’s own academic research has focused on making those systems easier for doctors to understand. He built an AI model that analyzed breast tissue images by comparing new samples with examples of healthy and abnormal tissue. He said the model was designed to show which examples shaped its diagnosis, presenting information in terms familiar to pathologists rather than only in mathematical terms. Yet despite the progress he and other researchers have made in understanding how AI reaches its conclusions, Parvatikar said he wouldn’t rely on AI for medical diagnosis.
“I think I would use AI as like your third or fourth order diagnosis,” he said. Even then, he would take the result with “a grain of salt.”
Parvatikar said that 77% of the cases in the Cancer Genome Atlas, a major public cancer dataset, come from white patients. For Aditi Ravi Kumar, a Columbia University graduate student in biomedical engineering, that makes a model’s training data the first thing patients should know before trusting it. “If you’re coming in from a different continent, you cannot trust the diagnosis that AI will make,” she said.
Research has found disparities in how pathology AI systems perform across demographic groups. In a 2025 study, a Harvard Medical School team tested AI models on 37 cancer diagnostic tasks, including tumor detection. In 29% of those tasks, the models performed significantly worse for some groups than others.
“That figure describes the proportion of tasks in our evaluation that showed statistically significant performance differences across demographic groups,” said Kun-Hsing Yu, the study’s senior author and an associate professor of biomedical informatics at Harvard Medical School. “It does not mean that 29% of patients were misdiagnosed.” He said a model can perform well overall while still being less reliable for some populations.
During the Q&A, audience members pressed Parvatikar on biased training data. He said researchers are still working on the problem and argued that medical AI systems should be more transparent about their training data.
Still, he said, “For me, I just care about my health and my diagnosis.”