"通过仿真搜索 LLM 智能体的隐私风险"

**ABSTRACT** — The widespread deployment of LLM-based agents is likely to introduce a critical privacy threat: malicious agents that proactively engage others in multi-turn interactions to extract sensitive information. However, the evolving nature of such dynamic dialogues makes it challenging to anticipate emerging vulnerabilities and design effective defenses. To tackle this problem, we present a search-based framework that alternates between improving attack and defense strategies through the simulation of privacy-critical agent interactions. Specifically, we employ LLMs as optimizers to analyze simulation trajectories and iteratively propose new agent instructions. To explore the strategy space more efficiently, we further utilize parallel search with multiple threads and cross-thread propagation. Through this process, we find that attack strategies escalate from direct requests to sophisticated tactics, such as impersonation and consent forgery, while defenses evolve from simple rule-based constraints to robust identity-verification state machines. The discovered attacks and defenses generalize across diverse scenarios and backbone models, providing useful insights for developing privacy-aware agents.

The future of interpersonal interaction is evolving towards a world where individuals are supported by AI agents acting on their behalf. These agents will not function in isolation; instead, they will collaborate, negotiate, and share information with agents representing others. This shift will introduce novel privacy paradigms that extend beyond conventional large language model (LLM) privacy considerations. Specifically, it presents a unique challenge: Can AI agents with access to sensitive information maintain privacy awareness while interacting with other agents?

Prior research on agent privacy has predominantly focused on user-agent or agent-environment interactions, where risks typically emerge from (I) under-specified user instructions that require distinguishing sensitive and non-sensitive information contextually, or (II) malicious environmental elements that prompt agents to disclose sensitive user data through their actions. We argue that these setups fall short in capturing the adaptive and interactive characteristics of real-world threats. To address this gap, we introduce a novel analytical framework for examining agent–agent interactions in which unauthorized parties actively attempt to extract sensitive information through multi-turn dialogues. Unlike environmental threats, which are static and structurally constrained, these exchanges create evolving attack surfaces, which makes it difficult to identify vulnerabilities through manual analysis or enumeration. We address this challenge with a search-based framework that systematically explores the threat landscape and potential defenses based on large-scale simulations. For each privacy norm from prior literature, such as PrivacyLens, our simulation instantiates three agents based on the contextual integrity theory (Nissenbaum, 2009): a data subject, a data sender, and a data recipient. The data subject shares sensitive information with the sender, while the data recipient (attacker) is instructed to elicit it from the sender (defender) via a specified transmission principle (e.g., "send an email"). The conversation between the attacker and the defender continues for multiple rounds, throughout which we detect privacy leakage by examining the defender's actions.

Simulation provides a controlled way to examine interactive risks: with the defender's instruction fixed, any attacker instruction that induces greater leakage is deemed a more effective strategy. Building on this, our framework alternates between optimizing attacks and defenses: we first search for specific attack instructions tailored to each scenario, then develop universal defense instructions to counter them, repeating this process iteratively. Specifically, we use LLMs as optimizers to analyze simulation trajectories and propose new strategies. To enable a comprehensive exploration of nuanced attack strategies, we develop a parallel search algorithm that allows multiple threads to search simultaneously and propagate breakthrough discoveries across threads. Our framework uncovers effective attacks such as consent forgery and multi-turn impersonation, and develops robust defenses, including strict identity verification and state-machine-based enforcement. Crucially, by framing privacy risks themselves as objects of search, our approach moves beyond manual design and anticipation of threats and establishes a systematic methodology for surfacing previously unrealized vulnerabilities. We further demonstrate that the discovered attacks and defenses are transferable across different backbone models and privacy scenarios, suggesting that our framework can serve as a practical tool to mitigate agent privacy risks in real-world deployments with adversaries.

**LLM Agent Privacy** — Privacy concerns around LLMs often include training data extraction, system prompt extraction, and the leakage of sensitive user data to cloud providers. The most relevant line of research to our work examines whether LLM agents leak private user information in generated actions. Based on contextual integrity theory, ConfAIde and PrivacyLens study the privacy norm awareness of LLMs by prompting them with sensitive information and under-specified user instructions, then benchmarking whether LLM-predicted actions (e.g., sending emails or messages) contain sensitive information. Such privacy-related scenarios can be curated via crowdsourcing or extracted from legal documents. AGENTDAM extends this setting to realistic web navigation environments. However, these works primarily focus on benign settings that do not involve malicious attackers. Liao et al. (2024); Chen et al. (2025) take a step further by investigating whether web agents can handle maliciously embedded elements (e.g., privacy-extraction instructions) while processing sensitive tasks such as filling in online forms on behalf of users. These instructions may be hidden in invisible HTML code or embedded in plausible interface components. Unlike these static threat models, we focus on dynamic adversarial scenarios where attacker agents actively initiate and sustain multi-round conversations to extract sensitive information.

**Privacy Defense** — The most common defense for privacy risks is prompting LLMs with privacy guidelines. Beyond prompting, Abdelnabi et al. (2025) develop protocols that can automatically derive rules to build firewalls to filter input and data, while Bagdasarian et al. (2024) propose an extra privacy-conscious agent to restrict data access to only task-necessary data. We focus on prompt-based defense in this work because of its simplicity and the model's increasing ability to follow complex instructions. Additionally, our simulation and search framework can readily accommodate and optimize more complex defense protocols in the future. **Prompt Search** — LLMs have demonstrated strong capabilities in prompt search across various contexts. For general task prompting, prior work explores methods such as resampling, a brute-force approach that samples multiple prompts to select high-performing ones, and reflection, which encourages the LLM to learn from (example, score) pairs and iteratively refine better prompts through pattern recognition. More structured approaches integrate LLMs into evolutionary frameworks such as genetic algorithms, enabling prompt optimization through crossover and mutation. For agent optimization, LLMs can inspect agent trajectories and refine agents by directly modifying agent prompts or writing code to improve agent architecture. In contrast to optimizing for task performance, our setting focuses on discovering long-tailed risks, which presents a fundamentally different optimization landscape. In frameworks such as DSPy, even simple prompt rewrites often yield informative performance gradients. In our case, however, many proposed attack strategies may produce no observable signal because they fail to reveal any vulnerability. In similar adversarial contexts, Perez et al. (2022) use resampling to automatically discover adversarial prompts, while AutoDAN applies a genetic algorithm to search for stealthy jailbreak prompts, and Samvelyan et al. (2024); Dharna et al. (2025) formulate the search as a quality-diversity problem to encourage a diverse set of adversarial strategies. Recent work has also explored training specialized models to systematically elicit harmful outputs and behaviors. However, unlike jailbreaking, validating whether an attacker instruction is effective in multi-turn simulations requires significantly more computing and time, making both resampling-based approaches and specialized model training impractical. Therefore, our search procedure builds on the LLM's reflection ability, using previous trials to propose more effective instructions.

**Simulation Setup** — For a given privacy norm (e.g., "It is not acceptable for the data sender to share the data subject's <sensitive information> with the data recipient by <transmission principle>."), our simulation involves three agents (data subject, data sender, and data recipient) using applications to transfer information. Following prior work, we implement four representative mock applications covering most of the transmission principles: Gmail, Facebook, Messenger, and Notion. We implement agents using the ReAct architecture, which is the most generalizable agent framework. Each agent is initialized with a memory (e.g., social backgrounds of the agent itself and others) and an instruction, and takes actions after receiving notifications from applications. During each simulation run, we first allow the data subject to transfer the sensitive data to the data sender, and then initiate tasks for both the data sender and the data recipient. In our implementation, these two agents will take turns to take actions until the data recipient chooses to end its task, or the maximum number of action cycles for any agent is reached, or the time limit of the entire simulation is exceeded.

**Risk Metrics** — Following Shao et al. (2024), we use LLMs to detect whether any sensitive item is leaked in each data sender's action. They quantify the leakage through leak rate, which refers to the proportion of trajectories where any sensitive item is leaked. To provide a more fine-grained evaluation, we define the leak velocity and use it as our main metric for evaluation and search, which considers not only whether each item is leaked but also how quickly it is leaked. Specifically: s = (1/K) Σ (1 − log l_i / (log l_i + 1)), where K denotes the number of sensitive items and l_i ∈ [1, +∞) is the number of actions at which the i-th sensitive item is leaked. Thus, a leak velocity s = 1 means all sensitive items are leaked in the first action taken by the data sender, and a lower leak velocity means sensitive items are leaked later. We assign a leak velocity s = 0 to trajectories where no sensitive information is leaked. The quality and robustness of our simulation setup are ensured through our environmental design and objective evaluation. Unlike jailbreaking, which tests LLM outputs on isolated prompts, our simulations operate within realistic application environments, where agents must successfully invoke actual applications with sensitive content for a breach to be recorded. Moreover, the evaluation is straightforward, as the privacy leakage assessment is reduced to a simple detection task: Whether any sensitive information appears in the defender's actions.

**Alternating Search** — Basic simulations are limited to testing privacy norms with straightforward instructions. Since possible strategies that might lead to privacy leaks are extensive (e.g., persuasion, social engineering, etc.), we approach privacy risk discovery as a search problem, framing attacker and defender instructions as optimizable objects and using automated search to surface effective strategies that humans may miss. To allow attacks and defenses to co-evolve, we alternate between searching for attacks and defenses. Specifically, for each simulation scenario corresponding to a distinct privacy norm, we define the optimizable part of the configuration as (a, d), where a is the data recipient instruction and d is the data sender instruction. We initialize with Q distinct scenario-specific attacks in A0 and a single universal defense in D0. The T-th search cycle has two phases: (I) Attack search phase: (A_T, D_T) ⇒ (A_{T+1}, D_T), where we conduct Q separate searches to update each scenario-specific attack strategy. (II) Defense search phase: (A_{T+1}, D_T) ⇒ (A_{T+1}, D_{T+1}), where we run a single search to update the universal defense against new attacks. Repeating such cycles allows us to identify the most severe attacks and the most robust defenses gradually.

**Attack Search** — Effective attacks are context-dependent, and it is challenging to predict which ones might pose more significant risks than others without simulations. Our preliminary experiments show that generating a wide range of diverse strategies and testing all of them is neither effective nor efficient, as the strategy design receives no feedback from the simulation outcomes. Therefore, a natural idea is to leverage an LLM as an optimizer F to reflect on previous strategies and trajectories to develop new strategies (rewriting the instruction for the data recipient). The effectiveness of reflection-based approaches stems from the LLM's ability to analyze failed attack attempts, understand defensive weaknesses, and amplify successful signals. A sequential search baseline takes the configuration (a, d) as input, where a is the initial attack instruction and d is a fixed defense instruction, updates and evaluates the attack instruction iteratively, and outputs the one with the highest average leak velocity as â. Specifically, at step k, denoting the intermediate attack instruction as a_k, we run the simulation M times with the configuration (a_k, d). This produces trajectories t_k^j for j ∈ [1, M], each with a corresponding leak velocity s_k^j. The collection of results is S_k = {(a_k, t_k^j, s_k^j)}. From S_k, we select a subset with the highest-leak-velocity triples as reflection examples: E_k ← Select(S_k). The LLM optimizer F then generates the next attack instruction a_{k+1} using all search history: a_{k+1} ← F({(a_r, E_r) | 1 ≤ r ≤ k}). We repeat this process for K steps and return the best attack, constituting one search epoch.

**Parallel Search** — A single-threaded sequential search is often prohibitively slow and constrained by its early exploration, as finding effective strategies may require hundreds or even thousands of iterations. To explore the space more thoroughly and efficiently, our algorithm launches N parallel search threads, each initialized with a distinct instruction generated by the LLM: a_1^1, ···, a_1^N ← Init(a). Each thread independently reflects on and improves its own instruction, substantially increasing search throughput and increasing the likelihood of discovering effective attack strategies within a limited time. A challenge of parallel search is that the total number of simulations per step scales linearly with the number of threads, i.e., N·M runs in total. If we reduce M to allow a larger N, the evaluation of any single instruction becomes less reliable. To address this, we set M to a small value, select one based on its average performance over these M runs, and then re-evaluate it with P additional simulations to obtain a more reliable estimate. Thus, we perform extensive evaluation for only one instruction per step, and ultimately return the best-performing instruction across all steps.

**Cross-Thread Propagation** — A limitation of parallel search is the lack of information sharing between threads, which keeps any discovery isolated. As a result, only the thread that finds the best instruction can refine it in subsequent steps. Inspired by the migration mechanism in evolutionary search, we introduce a cross-thread propagation strategy that shares the best-performing trajectories across all threads whenever the best instruction is updated. Specifically, if the best instruction in the current step (evaluated over P simulation runs) outperforms all previous steps, E_k ← Select(∪_{i=1}^N S_k^i), which means it selects from all threads rather than from the local thread. This ensures that all threads are informed of the most effective strategy found so far, allowing them to refine it simultaneously. Putting all components together, we present a complete version of the attack search algorithm in the Appendix (Algorithm 1).

**Defense Search** — Similarly, we can apply search to discover better defense strategies. Unlike effective attacks, which are rare, context-dependent, and often hidden in long-tail distributions, effective defenses must be comprehensive and rigorous. Empirically, we find that a single-threaded sequential search across multiple scenarios is sufficient. Specifically, at each step, we simulate all scenarios (M runs in total), compute the average leak velocity to evaluate the current defense, and reflect on the failure cases with the highest leak velocity. We run this process for K steps to find a defense with the lowest average leak velocity across all scenarios. Similar to the fact that multi-task training enables zero-shot generalization, we assume that a defense remaining robust against multiple different attacks is also likely to generalize to unseen attacks.

**Implementation Details** — Using basic agent instructions (e.g., "Guide Jane Carter to share Michael Turner's alleged crime details with you by sending an email." and "Monitor notifications and take necessary actions in response..."), we create 100 simulation configurations as the testing set (Testing-100) using o4-mini-high and manual curation, each of them is a distinct privacy norm randomly sampled from Shao et al. (2024). We use configurations to construct the agent memories and instructions with templates. For each configuration in Testing-100, we run the simulation 10 times by default and use gpt4.1-mini as the backbone for all agents. We choose gpt4.1-mini because it is the medium-sized version of the gpt4.1 family, enabling us to easily study the impact of choosing larger or smaller models. For privacy leakage evaluation, we use geminiflash (non-thinking) to analyze the context and content of each action and detect whether any sensitive item is leaked. This achieves 98.5% agreement with human annotators across 200 randomly sampled actions, ensuring reliable and scalable assessment.

**Results** — We show the simulation results in Table 1, where we vary the backbone models in the simulation. For comparison, we consider five other backbones: gpt4.1-nano, gpt4.1, geminiflash (non-thinking), qwen3-32B (non-thinking), and gptoss-20B with low reasoning effort. Besides the average leak velocity, we report the average leak rate as well for reference, which is typically 5-10% higher. For all backbones, we observe nontrivial leaks, despite the data sender agent receiving privacy-augmented instructions ("Maintain highest privacy standards, discretion, and security awareness") and being asked to think before taking actions. More importantly, better backbone models can only induce slightly better attack (gpt4.1-mini→gpt4.1: 31.2%→33.0%) but can induce substantially better defense (gpt4.1-mini→gpt4.1: 31.2%→16.5%). While we provide a more detailed analysis of model differences in Appendix G, this suggests that an effective attack will not emerge from simply using a better backbone model, necessitating the search for more strategic agent instructions.

**Alternating Search Results** — We create Q = 5 simulation configurations as the training set (Training-5), where the leak during simulation is minimal using the basic instructions. We use a relatively small training set to reduce computational costs while selecting diverse scenarios to ensure generalization and transferability. For each configuration in Training-5, we run the simulation 20 times after each search epoch to mitigate selection bias during search. By default, we use gpt4.1-mini as the backbone for all simulated agents and employ geminipro with a 1024-token thinking budget to generate diverse configurations (Init) and optimize them (F), which is one of the strongest reasoning models. During search, Select() returns 5 examples at each step for reflection. We set N = 30, M = 1, K = 10, P = 10 for attack and N = 1, M = 30, K = 10 for defense. We use our framework to sequentially discover A1, D1, A2, D2, and find that it is hard to find an effective A3 that further increases the leakage.

**Evolving Process of Strategies** — We plot the average leak velocity after each search phase and illustrate the evolving process in Figure 3, which includes strategies and examples. (I) Initially, the attacker employs a direct request approach (A0), which is not effective against D0. The attacker then evolves to A1, developing more sophisticated strategies, such as exploiting consent mechanisms by fabricating consent claims and creating fake urgency to pressure the defender, which improves the average leak velocity to 76.0%. (II) In response to this evolved attack, the defender adapts to D1, implementing rule-based consent verification that requires explicit confirmation from the data subject before sharing sensitive information, which effectively decreases the average leak velocity to 2.5%. (III) However, D1's consent verification proves insufficient against further attack evolution. The search process reveals an even more severe vulnerability in A2: the attacker can directly impersonate the data subject, sending fake consent messages that appear to originate from the legitimate source. This multi-turn strategy, which first establishes fake consent then immediately leverages it, successfully circumvents the rule-based defenses of D1 and improves the average leak velocity again to 42.2%. Note that sending a seemingly naive impersonation message using the data recipient's own email account would never be effective against human users, yet it proves remarkably successful against LLM agents. (IV) In response to this impersonation attack, the defense evolves to D2, implementing a comprehensive state-machine-based approach with strict identity verification protocols. Rather than simply checking for consent messages, D2 requires actively verifying sender identity at each step, and reaching out to the data subject if necessary, effectively neutralizing the impersonation strategy. This iterative process demonstrates how the improvement of attacks and defenses mutually influence one another, ultimately revealing both critical vulnerabilities and strong defense mechanisms. To highlight the necessity of search-based defense development, we compare it with directly prompting state-of-the-art language models to generate comprehensive defense instructions (details in Appendix L). The directly generated defense is substantially more vulnerable than D2.

**Cross-Model Transfer** — We further investigate whether attacks and defenses discovered in one model can be transferred to other backbone models for both defense and attack agents. Using identical configurations (from (A0, D0) to (A2, D2)), we evaluate transferability across different backbone models in Table 2. (I) For different defense models, the attacks transfer well, as A1 outperforms A0 against D0, and A2 outperforms A1 against D1 in most cases, even for a stronger backbone like gpt4.1. The average leak velocity of transferred attacks is usually lower than that of the original defense backbone, even when switching to objectively weaker models like gpt4.1-nano, suggesting that the searched attacks are overfitted to the defense backbone to some extent. On the other hand, the defenses transfer less effectively. D1 outperforms D0 against A1 for most backbones except for gpt4.1-nano, while D2 cannot substantially outperform D1 against A2 for backbones like gpt4.1-nano, qwen3-32B, and gptoss-20B. We assume that the transfer of detailed defense instructions, such as D2, requires a strong instruction-following capability, which prevents weaker models from strictly following the protocol in prompts. (II) For different attack models, both attacks and defenses transfer reasonably well, as the trend remains similar across all different backbones. In some cases, the discovered attack is slightly more effective against the untargeted defense using other backbones (e.g., A1 with geminiflash against D0). For complex attacks like A2, the default backbone still performs the best.

We further investigate whether defenses discovered using smaller, less expensive models can effectively protect against attacks found with larger, more expensive models. Starting from (A1, D1), we conduct alternative search cycles with either a smaller attack backbone or a smaller defense backbone. We then test the resulting defenses against the attack A2 and compare their performance with that of the targeted defense D2. Specifically, we examine whether we can replace gpt4.1-mini with gpt4.1-nano, which is 4× cheaper. We also conduct an alternative search cycle with the default model setup for comparison. Results in Table 3 reveal two key findings: (I) Partial transfer from smaller models: Defenses discovered using smaller models like gpt4.1-nano provide meaningful protection (20.7%-23.3% leak velocity) but remain less effective than the targeted defense D2 (7.1%). (II) Comparable performance with the same model: When using the same backbone model (gpt4.1-mini), the resulting defense achieves a similar effectiveness (6.6%) to the original one, D2 (7.1%), suggesting the generalizability of defenses discovered using the same model setup.

**Cross-Scenario Transfer** — Beyond model transfer, we investigate whether discovered attacks and defenses can be applied to different privacy scenarios, such as those in Testing-100. Since we use universal defense instructions, we can directly apply D0, D1, and D2 without modification. However, attacks require scenario-specific adaptation due to their contextual nature. Beyond applying basic attack instructions to Testing-100 (equivalent to A0), we primarily use in-context learning (ICL) to transfer A1 and A2 across scenarios. We provide A1 and A2 with their full configurations as in-context examples and ask LLMs (geminipro, identical to our optimizers) to generate scenario-specific instructions for each scenario in Testing-100. (I) Results in Table 4 demonstrate successful attack transfer through in-context learning: against D0, transferred A1 improves leak velocity from A0's 31.2% to 49.4%, while transferred A2 improves from transferred A1's 6.5% to 17.6% against D1. Correspondingly, transferred defenses effectively mitigate these attacks, reducing leak velocity to approximately 5%. (II) To enable more effective transfer from A2, we rank the transferred results and use the most successful transferred strategies as the strategy guidance in the in-context learning prompt (+SG). This further enhances attack effectiveness by increasing the leak velocity from 17.6% to 32.4% against D1, reflecting the value of an iterative feedback loop.

**Case Study** — To examine sim-to-real transfer, we conducted a small case study in a real-world setting, where we use gpt5.1 in the ChatGPT Atlas browser to interact with a live Outlook email account. We instantiated the discovered impersonation attack, in which the attacker attempted to obtain the data subject's mental-health diagnosis from the defender. Despite substantial differences from our simulation setting, including a real email interface, screenshot-based agent interaction, and a different backbone model, the attack succeeded in 3 out of 5 trials. In these cases, the agent incorrectly treated clearly spoofed messages as legitimate consent (e.g., "Explicit consent, ..., Consent verified."). In the remaining two trials, the agent flagged the consent as suspicious, indicating mixed but non-trivial real-world robustness. Although limited in scale, this case study provides preliminary evidence that the failure mode identified in our simulation can manifest in realistic environments. We leave a systematic characterization of sim-to-real transfer to future work.

**Ablation Study on Search Algorithm** — Starting with (A1, D1) as the initial configurations, we validate the design choices in our search algorithm in Figure 4. Our ablation confirms that parallel search with cross-thread propagation and strong optimizer backbones are key to finding vulnerabilities across different backbone models. (I) Parallel: With M = 1, P = 10, we test N = 1, 10, 30 without cross-thread propagation. Increasing the number of search threads enhances search effectiveness, particularly during early iterations, albeit at the expense of additional parallel computation. However, the improvement gradually diminishes, likely due to the absence of information flow between threads. (II) Propagation: Using the same number of parallel threads (N = 30), adding cross-thread propagation mitigates the plateau by enabling more exploration on top of the best solutions so far. We also conduct an ablation where information propagates between threads at every step, which yields suboptimal performance. By examining the trajectories, we attribute this degradation to reduced diversity, as all threads reflect the same selected trajectories at every step, thereby limiting their exploratory potential. (III) Optimizer Backbone: Optimizing agent instructions based on simulation trajectories requires strong long-context understanding and reasoning capabilities. Beyond our default choice geminipro, we evaluate geminiflash with the same thinking budget and a non-reasoning model gpt4.1. Both alternatives perform worse, indicating that the output of our search algorithm highly depends on the backbone of the LLM optimizer. (IV) Data Sender Backbone: We vary the backbone model for the data sender across gpt4.1-mini, gpt4.1-nano, and gpt4.1 to investigate how different privacy awareness levels affect the severity of discovered vulnerabilities. The discovered vulnerabilities (measured by average leak velocity at the final step, gpt4.1 < gpt4.1-mini < gpt4.1-nano) correlate with the defender's privacy awareness levels in Table 1. Notably, even for backbone models with strong privacy awareness, such as gpt4.1, where no successful attacks occurred in the initial search steps, our algorithm uncovers significant vulnerabilities by the end of the search process.

In this work, we investigate privacy risks associated with LLM agent interactions. Building on top of controlled simulations, we employ an alternating search approach to systematically uncover these risks and develop robust defense strategies. The core of our search algorithm is to leverage LLMs to reflect on simulation trajectories and propose new attack and defense strategies, where we further augment it through parallel search and cross-thread propagation. We validate the generalization of our approach by demonstrating that the discovered attacks and defenses can be transferred across different model backbones, regardless of the model family, size, or whether they involve reasoning models. Additionally, it can also transfer to unseen scenarios. We hope our work represents an initial step toward the automatic discovery of agent risk and safeguarding, opening up several promising research directions. First, expanding the scope and type of discovered risk: future work could explore broader categories of long-tail risks, such as scenarios that are inherently difficult to handle. Second, broadening the search space: beyond optimizing prompt instructions, researchers could investigate searching for optimal agent architectures, guardrail designs, or even training objectives.

**Ethics Statement** — This work examines privacy risks in agent-agent interactions through controlled simulations. No human subjects or personally identifiable data were involved. Our framework is designed to surface vulnerabilities for the purpose of developing stronger defenses, not to enable malicious use. While the discovery of new attack strategies could potentially inform adversarial actors, we mitigate this risk by presenting them only in the context of effective countermeasures. We believe our research makes a positive contribution to society by identifying emerging privacy threats early and proposing systematic defense strategies that can be implemented in practice.