Life Off-Label

In 2022, my advisor and I published a paper in Alzheimer Disease and Associated Disorders asking a pointed question: How often are older adults with Alzheimer’s disease prescribed antipsychotic medications in ordinary outpatient care? We wanted to know because antipsychotics carry an FDA black box warning for increased mortality in elderly patients with dementia. Additionally, the Beers Criteria – the standard reference for potentially inappropriate prescribing in older adults – flags them as potentially harmful in this population.

Using the CDC’s National Ambulatory Medical Care Survey (NAMCS), a nationally representative sample of outpatient office visits, we found that visits by older adults with Alzheimer Disease had 6.81 times higher odds of an antipsychotic prescription than visits without it. Furthermore, despite the very serious black box warning and guidelines recommending against their use, almost 10% of all visits among people with Alzheimer’s disease included an antipsychotic medication.

These results were pretty interesting, but getting those numbers was not fast. NAMCS is a public-use file, but “public” does not mean “easy.” You learn what the variables mean from a separate documentation file. You define the population, apply inclusion/exclusion criteria, and hand-code which drugs count as antipsychotics. And, because NAMCS is a complex survey with weights, you fit the models with survey-specific methods. It was boring, tedious, and frustrating, but a valuable learning opportunity.

I’ve written before that my AI tools are basically grad students. This time, I gave the good one a comps exam.

I’ve been playing around with Claude Science over the past couple of weeks and have been impressed. An idea occurred to me that made me think, Let me really test how good this is.

This is the exact prompt:

Just curious of something. This is just for funsies, but this is a study I did with NAMCS data a while ago. I wanted to see if you are able to replicate it? https://pubmed.ncbi.nlm.nih.gov/35700324/

That’s it. I did not hand over my data. I did not share my code. I did not walk it through my methods. I gave it a citation on Pubmed to see what it could do.

Y’all… Claude went and pulled the raw NAMCS files off the CDC’s server. It found the documentation, parsed the data, and reconstructed the variables. It worked out the drug classification hierarchy on its own. It checked its logic against the official documentation as it went. It applied the survey weights and ran a survey-weighted logistic regression.

This first round was pretty impressive on its own. Claude got results pretty similar to mine, but I wanted to see if we can do even better.

Round 2

I knew Claude didn’t have access to the manuscript, so I was curious how his methods in the first round differed from mine. I gave Claude some more detail of our actual methods so that he could more truly replicate my study. He did his thing, re-ran the numbers, and got much closer to my actual results. There were a few key differences from the first run:

• Case definition. NAMCS has a checkbox coded “Alzheimer’s disease/Dementia.” Claude used it directly. Ironically, back when I was a grad student myself, I had also wanted to use it to take the easy way out. However, after talking with my wise advisor, we had defined Alzheimer’s using diagnosis codes from the visits alongside that variable. We also excluded certain visits such as those with a diagnosis for Lewy Body Dementia.
• Exclusions. In addition to above, we removed visits involving cancer and psychoses as the benefit of antipsychotics in those cases often outweigh the risk. The AI’s first pass didn’t do this.
• Drug coding. My advisor and I are pharmacists and we coded our antipsychotic list manually. The AI used the dataset’s built-in therapeutic categories. It is a reasonable, more automated choice, but it lacks clinical judgment.
• Comorbidity coding. We adjusted for comorbidity with a modified Deyo Charlson Comorbidity Index, hand-built from the diagnosis codes. It’s more tedious, but a validated approach to measuring comorbidities in our field. Claude used the dataset’s raw count of chronic conditions. It’s not technically wrong (and I think I also initially wanted to do this to take the easy way out), but counting conditions is not the same as weighting them.
• And there were a few others, but you made it this far without falling asleep.

After Claude ran round 2, the replicated results got even closer:

• My weighted Alzheimer’s population: 15,471,125 visits. Claude’s replication: 15,585,727. A difference of 0.7%.
• My raw Alzheimer’s visit count: 461. Claude’s count: 466.
• Odds of polypharmacy: 1.15. Claude’s odds: 1.17.
• My adjusted odds of antipsychotic medication: 6.81. Claude’s: 5.45. (both statistically significant)

AI is Getting Good, Very Good

Consider me thoroughly impressed. I didn’t do this just to see if my study held up (but I am glad it did). It’s what this tool did, unsupervised from just a PubMed link, that had me blown away. With Claude Science, you can watch it go through the files, write the Python code, describe its thought process, and even double-check its own work in real time. It was genuinely interesting to watch.

But it also scares me a bit. If AI can rebuild a published study in such a short amount of time with minimal prompting, then someone can also produce a study in the same time frame. You can throw out an idea, let it take the reins, and walk away with something wearing all the costume of rigor: plausible methods, survey weights, confidence intervals, a clean table, etc. That study will be genuinely hard to distinguish from one where a researcher spent many hours hand-coding and parsing the literature to inform each decision.

The first run wasn’t inherently wrong, but it didn’t quite know the nuance. It didn’t know why a Charlson Index is considered a gold standard for assessing comorbidities instead of counting them. It also didn’t have the clinical rationale for pulling cancer visits out of the sample. Many people would read that first analysis and think it’s fine. However, someone with expertise in the field would have questions, and those questions are exactly why we have peer review.

That said, there’s a selfish upside, too. My ADHD brain comes up with study ideas all the time, but many of them never move past the idea phase. Part of the problem is figuring out feasibility. Is there a true literature gap? Does the data have what I need to answer my research question? At least with things like publicly available data sets, Claude can do some of that preliminary legwork for me.

Claude passed the exam and we love that for him, but his advisor has mixed emotions. 

---

Original study: Tidmore LM, Skrepnek GH. A National Assessment of Alzheimer Disease and Antipsychotic Medication Prescribing Among Older Adults in Ambulatory Care Settings. Alzheimer Dis Assoc Disord. 2022 Jul-Sep 01;36(3):230-237. doi: 10.1097/WAD.0000000000000509. Epub 2022 Jun 7. PMID: 35700324.