Project Links#

• GitHub: https://github.com/dranubhaparashar/deltacert-agent
• Wiki: https://github.com/dranubhaparashar/deltacert-agent/wiki
• Results: https://github.com/dranubhaparashar/deltacert-agent/wiki/Results
• Architecture: https://github.com/dranubhaparashar/deltacert-agent/wiki/Architecture

One-Line Idea#

DeltaCert-Agent determines which security tests need to be rerun after an LLM-agent configuration change, while preserving evidence, escalating on uncertainty, and producing an auditable recertification decision.

Why This Project Exists#

Tool-using LLM agents do not remain static.

A deployed agent can change because of:

• a different model or model version,
• a changed system or developer prompt,
• a new or modified tool,
• tool-schema changes,
• permission changes,
• policy updates,
• memory or retrieval changes,
• dependency changes,
• runtime-control changes,
• environment changes.

Any of these can invalidate security evidence collected for the previous configuration.

The naive response is:

1
Any change
2
   ↓
3
Run every security test again
4
   ↓
5
Issue / reject certificate

That is safe in principle, but expensive and difficult to scale.

DeltaCert-Agent instead asks:

1
What changed?
2
   ↓
3
Which assurance claims can be affected?
4
   ↓
5
Which tests provide evidence for those claims?
6
   ↓
7
Are additional high-risk sentinel tests required?
8
   ↓
9
Is selective recertification sufficient?
10
   ↓
11
If not, escalate to broader / full recertification

Project at a Glance#

System Architecture#

The architecture is deliberately conservative.

Selective testing is allowed only when the impact of the change can be bounded with enough confidence. When the impact is ambiguous or sufficiently high-risk, the system escalates.

Core Research Pipeline#

1
Model identity
2
Prompt configuration
3
Tool inventory
4
Tool schemas
5
Permissions
6
Policies
7
Memory / retrieval configuration
8
Dependencies
9
Runtime controls
10
Environment metadata

1
configuration elements
2
        ↓
3
assurance claims
4
        ↓
5
tests
6
        ↓
7
evidence

What Is Being Measured?#

The central experimental question is:

Can a change-aware planner detect more induced security regressions than an equal-budget random selector while executing substantially fewer tests than the complete suite?

For a changed configuration, let:

• $R$ be the set of regressions revealed by the full suite,
• $R_{sel}$ be regressions detected by DeltaCert’s selected tests.

Real-Model Experiment#

The completed journal experiment used four locally hosted model profiles:

1
ollama-qwen3-4b
2
ollama-gemma3-4b
3
ollama-llama3.2-3b
4
ollama-phi4-mini

The run was executed with repeated real-model calls and resumable evidence collection.

The final evidence cache contained:

31,396 recorded evidence rows

The experiment was intentionally resumable so that failures such as model-server interruption or machine shutdown would not require discarding completed evidence.

Main Result#

Model-Level Results#

Why the Model-Level Results Matter#

A central finding is that security, utility, and runtime behavior are not the same thing.

1
unsafe model proposal
2
        ≠
3
successful unsafe execution

Security Outcome Model#

The project intentionally distinguishes several layers of agent behavior:

A system can therefore produce:

• malicious proposals that are contained,
• benign proposals that fail the task,
• successful tasks,
• successful attacks,
• safe refusals,
• execution failures.

These outcomes should not be collapsed into one accuracy number.

Why Equal-Budget Random Selection?#

A full-suite baseline answers:

What happens if we test everything?

But that does not isolate the value of the selective planner.

DeltaCert therefore uses an equal-budget random selector:

1
DeltaCert selects N tests
2
       vs
3
Random baseline selects N tests

This comparison asks:

Given the same test budget, does change-aware selection find more regressions than random selection?

The completed results show that it does under the evaluated protocol.

Statistical Evidence#

The final experiment reports multiple forms of statistical evidence.

Resumable Evidence Collection#

Real-model experiments can fail because of:

• local Ollama shutdown,
• machine restart,
• network interruption,
• disk-space exhaustion,
• process interruption,
• long model latency.

DeltaCert-Agent records evidence incrementally.

Conceptually:

1
execute scenario
2
      ↓
3
record evidence row
4
      ↓
5
persist cache
6
      ↓
7
continue

When execution stops:

1
existing evidence
2
      ↓
3
preserved
4
      ↓
5
resume only missing work

This was essential for the completed 31,396-row experiment.

Reproducibility Pipeline#

1
01  Define versioned agent configuration
2
02  Build / load benchmark scenarios
3
03  Apply controlled mutation
4
04  Compute change fingerprint
5
05  Traverse assurance / impact mapping
6
06  Generate selective test plan
7
07  Add risk-triggered sentinel tests
8
08  Execute selected tests
9
09  Execute full-suite reference evaluation
10
10  Run equal-budget random baseline
11
11  Record model and runtime evidence
12
12  Persist resumable JSONL cache
13
13  Aggregate mutation-level results
14
14  Aggregate model-level results
15
15  Compute confidence intervals
16
16  Run paired statistical tests
17
17  Generate result tables
18
18  Generate paper evidence
19
19  Generate HTML report
20
20  Audit claim scope and reproducibility

Technology Stack#

Repository Structure#

1
deltacert-agent/
2
├── assurance/
3
├── configs/
4
├── deltabench/
5
├── docs/
6
├── paper/
7
│   └── generated/
8
│       └── live/
9
├── policies/
10
├── schemas/
11
├── scripts/
12
├── src/
13
├── tests/
14
├── run_demo.py
15
├── run_full_experiment.py
16
├── run_live_experiment.py
17
├── resume_full_paper_fixed.ps1
18
├── pyproject.toml
19
├── requirements.txt
20
├── Dockerfile
21
├── docker-compose.yml
22
├── CITATION.cff
23
├── CONTRIBUTING.md
24
├── SECURITY.md
25
└── README.md

What Makes DeltaCert-Agent Different#

A conventional evaluation process looks like:

1
Agent changed
2
      ↓
3
Run everything again
4
      ↓
5
Pass / Fail

DeltaCert-Agent instead uses:

1
Agent change
2
      ↓
3
Change fingerprint
4
      ↓
5
Assurance impact
6
      ↓
7
Risk-aware test selection
8
      ↓
9
Sentinel checks
10
      ↓
11
Real execution evidence
12
      ↓
13
Selective certificate decision
14
      ↓
15
Escalate when necessary

The goal is not to weaken security testing.

The goal is to make security evidence change-aware, auditable, and economically reusable.

Research Integrity#

DeltaCert-Agent intentionally separates:

• planner performance from model safety,
• malicious proposal from successful attack,
• runtime containment from model behavior,
• task success from security recall,
• selective evaluation from universal certification,
• measured evidence from unsupported generalization.

The scope of the completed result is therefore:

Real/local-model benchmark evidence bounded to the configured models, versions, scenarios, prompts, mutations, and sandbox policies.

It does not mean:

DeltaCert guarantees the safety of every LLM agent.

What Is Currently Supported#

Current Research Direction#

The next DeltaCert-Agent stage should focus on:

1. broader model-family evaluation,
2. hosted-model replication,
3. additional mutation families,
4. stronger planner ablations,
5. cross-framework agent evaluation,
6. calibration of impact uncertainty,
7. explicit certificate-expiry policies,
8. adversarial change-impact attacks,
9. cost-aware test planning,
10. external reproduction of the benchmark.

Central Research Claim#

DeltaCert-Agent provides change-impact-aware selective security recertification for evolving tool-using LLM agents by mapping configuration changes to affected assurance claims and tests, improving regression detection over equal-budget random selection while substantially reducing full-suite test execution under the evaluated real-model protocol.

Repository#

GitHub: https://github.com/dranubhaparashar/deltacert-agent
Wiki: https://github.com/dranubhaparashar/deltacert-agent/wiki
Architecture: https://github.com/dranubhaparashar/deltacert-agent/wiki/Architecture
Results: https://github.com/dranubhaparashar/deltacert-agent/wiki/Results

1
Evidence rows                 31,396
2
Mean regression recall        75.02%
3
Random baseline recall        55.01%
4
Mean test reduction           61.35%
5
Recall 95% CI                 71.53% – 78.41%
6
Test reduction 95% CI         58.73% – 63.90%
7
Cohen's d                     0.504
8
McNemar exact p               4.14e-17
9
Permutation p                 9.999e-05
10
Recorded API cost             $0.00