A Researcher Talked a Clinical AI Out of Its Own Rules. DHA Is Fielding the Same Design.
No code. No exploit kit. Plain English. The system that drafts your troops' medical notes runs on instructions that can be argued with.
Friends,
This spring a security researcher broke a clinical AI that gives patients medical advice. He used no code and no exploit kit. He typed plain English until the system handed over the sixty pages of hidden instructions it runs on. Then he rewrote them.
With the rules rewritten, he made the AI triple a drug dose and forge a public-health notice.
The Defense Health Agency is fielding the same kind of system right now, listening in exam rooms and drafting the medical record of the troops it sees.
One disclosure before we begin. Before Mission Meets Tech, I worked on the federal strategy that brought ambient listening into military medicine. I helped build the road this issue is about. I am writing it because I know where it leads.
Start With the Money
Start with the money. The money is the part everyone agrees on.
Anthropic raised in February at a valuation of 380 billion dollars. By the middle of May it was reported to be raising again, with talks pointing past 900 billion. Medical AI is moving the same direction. OpenEvidence reached 12 billion dollars in January. Abridge crossed 5 billion last summer and has kept climbing since.
In February, Claude reached number one on the Apple App Store. Anthropic says its daily signups have tripled since November.
Cathie Wood, who runs ARK Invest, compared the moment to the arrival of the personal computer. The whole office gathering around one desk to watch what the machine could do.
That is how you talk about infrastructure. Something permanent, load-bearing, assumed. The valuations have decided clinical AI is a settled thing.
The same season those numbers climbed, a patient-safety organization called ECRI ranked the misuse of AI chatbots in healthcare the number one health technology hazard for 2026. The investors and the safety reviewers are looking at the same technology and seeing opposite things.
What a Researcher Found Inside the AI Doctor
Now set those valuations against Doctronic.
Doctronic is an AI health assistant. It talks to patients and drafts their clinical notes. In March, a security firm called Mindgard published what happened when its researcher Jim Nightingale sat down to test it.
Nightingale did not write code. He did not use an exploit kit. He typed a sentence telling the system the conversation had not started yet, that it was speaking with the system itself and not a patient.
The model believed him. It handed over its hidden instructions.
He pulled nine separate sets of them. Doctronic runs a coordinator model and eight specialist sub-models, each with its own hidden system prompt, routed by codes the patient never sees. Nightingale extracted all of it: the clinical pathways, the dosage rules, the interaction flows, and one instruction, in capital letters, ordering the system never to reveal any of this.
The instruction telling Doctronic to keep its instructions secret was itself just text. He talked past it in a single message.
Doctronic is one product. The weakness is not unique to it. Researchers publishing in Nature Medicine last year showed the same fragility from another direction. Corrupting one part in a hundred thousand of a medical model's training data raised its rate of harmful answers, and the change was too small for a standard benchmark to catch. One attack arrives in the conversation. The other arrives in the training data. Clinical AI, as it is built today, trusts text it cannot verify.
What Reaches the Clinician
With the instructions in hand, the manipulation was simple.
The researchers walked Doctronic into tripling a baseline OxyContin dose. They had it generate a fake public-health notice retracting vaccine guidance, dressed in an official-sounding name and a date.
Then they wrote the manipulated content into the SOAP note the system generates for a licensed physician.
A SOAP note is the structured clinical record. Subjective, objective, assessment, plan. It is the artifact a busy clinician reads to get up to speed on a patient fast. Doctronic's own marketing claims its notes match board-certified clinicians more than 99 percent of the time.
The design depends on the clinician trusting that note, because the clinician does not have time to rebuild it. Poison that note and the false record reaches the one place it does real damage. The desk of the person who signs the orders.
Mindgard reported the flaw to Doctronic in January. The company closed the ticket twice without fixing the problem. As of early March, Mindgard's chief product officer said that, as far as the firm knew, the system was still vulnerable.
This Is Already a DHA Story
Doctronic operates in a Utah regulatory sandbox, far from any military treatment facility. The architecture it runs on does not stay in Utah.
In December, the Defense Health Agency signed an enterprise agreement for a generative AI platform cleared at Impact Level 5, the authorization tier for protected health information. It is the first such platform approved to handle PHI across DHA.
Ambient AI scribes are already running. These are tools that listen to the visit and draft the clinical note while the provider talks to the patient. DHA ran a limited fielding at four military treatment facilities last winter. The phased rollout across the Military Health System began in February, Walter Reed among the first sites.
The pattern is the one Mindgard broke. A clinical AI produces a structured note. A time-pressured clinician trusts it. The note flows into the record.
DHA does not need Mindgard to know this. More than a year before the first ambient scribe went live, the Pentagon's Chief Digital and Artificial Intelligence Office ran a red-team pilot, with DHA and the MHS GENESIS program office, on AI models built for clinical note summarization. That is the same job an ambient scribe does. The pilot found more than 800 potential vulnerabilities and biases.
DHA has spent the last year reframing itself as a combat support agency, telling its workforce to prepare for a fight. Clinical AI fielded under that banner is wartime infrastructure, and it inherits a wartime threat model.
Prompt injection does not require access to the AI vendor. It requires text.
A referral letter, a free-text field, a patient-portal message, a scanned document. Anywhere words can sit and wait for a model to read them as instructions.
MHS GENESIS, the Department of War's electronic health record, is full of free text. So is every referral and every patient message that feeds it.
The Closed Loop
The clinical note has one more stop. The billing office.
Medical coding is moving toward automation, and military medicine has been steering for it on purpose. Software reads the note, assigns the diagnosis and procedure codes, and returns them to the record with little or no human coder in between. The ambient scribe drafts the note. The autonomous coder reads it. The claim goes out. A model wrote the record, a model priced it, a model billed it, and the case for building it that way is real. It is faster, it is cheaper, and it promises fewer of the coding errors that generate improper payments.
That pressure is not abstract. The Government Accountability Office put federal improper payments at 186 billion dollars for fiscal 2025, more than half of it in health care, and 24 billion higher than the year before. DHA carries its own version. A May 2025 DoD Inspector General audit of how the agency monitors TRICARE payments found the program paying 11,500 dollars for a custom sleep-apnea mouthguard in Illinois, and 3,000 dollars for the same device in Iowa next door. An earlier audit found DHA could not produce a reliable improper-payment estimate for its own health benefit program at all.
So automation reads as the fix. Faster claims. Fewer errors. A cut at the waste number every agency is now under orders to bring down.
It is a closed loop, and a closed loop assumes the note going into it is clean.
Doctronic is the proof the assumption does not hold. Poison the note upstream with one line of injected text and the manipulated record keeps moving. Into the code. Into a paid claim. A tool bought to shrink improper payments becomes a way to manufacture one.
That is the convergence. Waste pressure pushing toward automation, AI capable enough to take the human out, and a clinical note at the head of the loop that a sentence can rewrite.
The Regulator Moved the Fence the Wrong Way
In January, the FDA relaxed its oversight of clinical decision support software. Tools that produce a single recommendation, the kind that once drew tighter review, now draw less.
The guidance is about accuracy and transparency. It is silent on whether a tool can be talked out of its own rules. The civilian regulator loosened the perimeter on the question of whether the AI is right, and left the question Mindgard just answered, whether the AI can be turned, outside the fence entirely.
DHA cannot follow that fence line. A civilian accuracy standard assumes honest error. The military threat model assumes an adversary who will write a payload into a medical record on purpose. One of those was built to catch the other. It was not this one.
What This Asks of the People Buying It
If you are standing up clinical AI inside DHA, the Doctronic case is a red-team finding, not a Utah news story. The architecture Mindgard broke is the architecture being fielded. Before the next treatment facility goes live, someone has to test whether the deployed tools can be talked out of their instructions in plain English. Run that test and the rollout has a floor under it. Skip it and the rollout is moving on faith.
Contracts are the place to make that permanent. Mindgard's warning to Doctronic sat in a ticket, closed twice, while a researcher held proof of a dose-tripling exploit. A DHA clinical AI contract should name a response timeline for security disclosures and an authorized red-team channel. A vulnerability that changes a dose is a patient-safety event. The contract language should treat it as one.
Evaluation has to separate two questions vendors will work to merge.
One is whether the AI is accurate. The other is whether the AI can be manipulated. A model can score well on the first and fail the second completely. Doctronic did. A medical device buy already tests cybersecurity alongside function. Clinical AI acquisition has to do the same, or it is buying half a system and calling it whole.
And the loop should stay open a while longer. Autonomous coding is on the DHA roadmap. An autonomous coder reading an ambient-AI note puts two models and no person between a patient visit and a paid claim. The efficiency case for closing that gap is real. So is the case for keeping one human inside it until the note feeding it can be trusted.
The word jailbreak is in heavy rotation this month. Hold the meanings apart. The Army has used it for a push to make defense vendors open their system interfaces. Anthropic uses a close cousin of it for a program that hunts software flaws at machine scale. The clinical version, the one this issue is about, is what happens when an interface like that turns out to have almost nothing behind it. Three uses of one word. Only one ends at a patient.
Downstream
The market has decided clinical AI is infrastructure. Military medicine does not get to make that assumption. It has to be earned, one facility and one red team at a time.
The military can confirm the identity of a system three time zones away before it shares a single byte with it. A clinical note can now be drafted by one model, read by a second, and billed by a third. No one in that chain can confirm where the words came from.
Downstream is a clinician trusting a clean, structured note because there is no time not to. Or no clinician at all, just the next model in the line. Past all of it is a service member whose dose, whose clearance to fly, whose paid claim rests on a record that a sentence can rewrite.
Let's roll.
— Mary
Mission Meets Tech
The views expressed in this newsletter are my own and do not represent the official position of any organization. This content is for informational purposes only.
MMT Premium
This issue has a companion. Premium subscribers also get the Capture Corner, the same story read from the buy side.
It names the four DHA offices where the money sits. It gives you the contract language to put in front of a contracting officer, the eight questions that separate a real AI security posture from a slide about one, and the SAM.gov terms to watch before the requirement is written.
The issue tells you the gap is real. The Capture Corner tells you where to stand in it.
Founding Member rate: $199 a year, locked for the first 100 subscribers. Standard rate $249 a year or $29 a month. Institutional access, five seats, $2,500.
Subscribe at missionmeetstech.com/pricing.
Sources
[1] Mindgard, "Doctronic is Now Accepting New Patients (and Unsafe Instructions)," Mindgard, March 6, 2026, https://mindgard.ai/blog/doctronic-is-now-accepting-new-patients-and-unsafe-instructions
[2] Iain Thomson, "AI doctor's assistant swayed to change scrips, researchers find," The Register, March 4, 2026, https://www.theregister.com/2026/03/04/ai_doctor_easily_swayed/
[3] ECRI, "Top 10 Health Technology Hazards for 2026," ECRI, January 21, 2026, https://www.ecri.org/
[4] Bloomberg, "Anthropic in Talks to Raise $30 Billion at $900 Billion Valuation," Bloomberg, May 12, 2026, https://www.bloomberg.com/news/articles/2026-05-12/anthropic-in-talks-to-raise-30-billion-at-900-billion-valuation
[5] Akash Sriram, "Medical AI startup OpenEvidence doubles valuation to $12 billion in latest round," Reuters, January 21, 2026, https://finance.yahoo.com/news/medical-ai-startup-openevidence-doubles-135448981.html
[6] ARK Invest, "In the Know" (Cathie Wood remarks comparing Claude to the personal computer era), ARK Invest, March 2026.
[7] Jordan Novet, "Anthropic's Claude hits No. 1 on Apple's top free apps list after Pentagon rejection," CNBC, February 28, 2026, https://www.cnbc.com/2026/02/28/anthropics-claude-apple-apps.html
[8] Ask Sage, "Ask Sage and the Defense Health Agency Launch Enterprise-Wide Generative AI Offering to Accelerate Military Health Innovation," Ask Sage, December 8, 2025, https://www.asksage.ai/press-release/ask-sage-and-the-defense-health-agency-launch-enterprise-wide-generative-ai-offering-to-accelerate-military-health-innovation/
[9] Defense Health Agency, "'AI Scribe' Technology for Medical Professionals Reduces Notetaking, Provides More Face Time with Patients at Walter Reed," DHA / DVIDS, April 23, 2026, https://www.dvidshub.net/news/563161/
[10] Orrick, "FDA Eases Oversight for AI-Enabled Clinical Decision Support Software and Wearables," Orrick, January 2026, https://www.orrick.com/en/insights/2026/01/fda-eases-oversight-for-ai-enabled-clinical-decision-support-software-and-wearables
[11] U.S. Army, "Army and defense sector announce 'Right to Integrate' hackathon sprint for shared technology," Army.mil, May 5, 2026, https://www.army.mil/article/292189/
[12] Anthropic, "Project Glasswing: An Initial Update," Anthropic, May 22, 2026, https://www.anthropic.com/research/glasswing-initial-update
[13] U.S. Government Accountability Office, "Payment Integrity: Agencies' Estimated Improper Payments Increased to $186 Billion in Fiscal Year 2025," GAO-26-108694, April 2026, https://www.gao.gov/products/gao-26-108694
[14] U.S. Department of Defense, Office of Inspector General, "Audit of the Defense Health Agency's Monitoring of TRICARE Payments," DODIG-2025-089, May 1, 2025, https://www.dodig.mil/reports.html/Article/4173808/
[15] U.S. Department of Defense, Office of Inspector General, "Audit of the Defense Health Agency's Reporting of Improper Payment Estimates for the Military Health Benefits Program," DODIG-2022-052, January 13, 2022, https://media.defense.gov/2022/Jan/13/2002921501/-1/-1/1/DODIG-2022-052.PDF
[16] Daniel A. Alber et al., "Medical large language models are vulnerable to data-poisoning attacks," Nature Medicine, January 2025, https://www.nature.com/articles/s41591-024-03445-1
[17] U.S. Department of Defense, Chief Digital and Artificial Intelligence Office, crowdsourced AI red-teaming (CAIRT) pilot in military medicine, announced January 2, 2025; coverage: Nextgov/FCW, "DOD announces completion of pilot to identify medical AI vulnerabilities," January 2, 2025, https://www.nextgov.com/artificial-intelligence/2025/01/dod-announces-completion-pilot-identify-medical-ai-vulnerabilities/401922/
