Privacy Leakage in Small Agentic Healthcare Models: An Empirical Analysis

The advent of agentic AI dramatically transforms how healthcare chatbots are designed and deployed. Since language models can now autonomously interact with external tools —such as search engines, third-party APIs, and databases— and given the proliferation of small, task-specific models, new opportunities emerge to improve response quality and provide more personalised and context-aware solutions.However, these capabilities also introduce critical challenges regarding privacy and security of sensitive data. Access to external resources may leak sensitive information through both direct and indirect pathways. Moreover, smaller models may exhibit reduced robustness in adhering to privacy guidelines via system prompts, such as avoiding transmission of sensitive data to external services. These concerns are particularly critical in healthcare, where sensitive patient information is routinely involved.To address this, this paper empirically investigates the privacy-preservation capabilities of agentic language models. We propose an evaluation pipeline to quantify leakage risks during tool interactions, comprising a synthetic dataset generator that emulates realistic healthcare scenarios and an LLM-as-a-judge assessment framework. Our results demonstrate that small agentic models, specifically Qwen-3 1.7B and 4B, fail to consistently enforce privacy guidelines, exhibiting high leakage rates (56% in the case of Qwen-3 4B) even when provided with strict system instructions. These findings underscore the urgent need for dedicated privacy-preservation mechanisms in small-scale agentic healthcare systems.