COPA: Continual Preference Optimization for Adaptive Prompt Injection Defense

Source Summary

What It Contains

The COPA preprint proposes continual preference optimization as a defensive paradigm against evolving prompt injection attacks, using GRPO-based optimization on newly observed attacks and margin-weighted experience replay to retain defenses against prior attack classes.

Extracted Claims

• The authors argue that existing prompt-injection defenses are predominantly static, relying on fixed alignment objectives or attack-specific filtering that requires redesign as new attack strategies emerge.
• COPA treats prompt-injection defense as a lifelong learning problem and incrementally incorporates feedback from newly observed attacks via GRPO-based optimization.
• Margin-weighted experience replay is used to retain defenses against prior attack classes while mitigating catastrophic forgetting and preserving general-purpose model capabilities.
• The paper reports that across lifelong prompt injection attack streams, COPA reduces attack success rate by up to 6.3x and 4.4x on average compared to state-of-the-art defenses.
• The contribution is a training-time alignment methodology rather than a runtime defense or detection mechanism.

Evidence Quality

Academic preprint on arXiv (cs.CR). Reported numbers are research claims pending independent replication and peer review. COPA is a model-training technique, so its operational relevance for Armorer operators is indirect: it informs model selection, alignment-pipeline design, and awareness of adaptive-adversary defense paradigms rather than providing a directly deployable runtime control.

Armorer Relevance

COPA is complementary to Armorer's runtime and operational controls. Operators cannot directly run COPA on a deployed model, but they can use COPA-style alignment evidence to evaluate model providers, to set expectations about how often models should be retrained against emerging attack classes, and to plan for continual adversarial evaluation rather than one-time alignment benchmarks. It also reinforces the case for layering training-time alignment with runtime screening, action gates, and prompt-injection regression testing.

Follow-Up

• Track whether COPA publishes implementation details, independent evaluations, or downstream model integrations.
• Compare COPA with Armorer runtime-monitoring, action-gate, and memory-screening controls as complementary defense layers.
• Monitor whether model providers adopt continual preference optimization as a built-in defensive technique for agent deployments.