One-Line Idea#

IncidentGraph reconstructs multi-camera incidents without hiding where evidence came from, which claims disagree, which observations are missing, and how uncertain the final reconstruction remains.

The central design principle is:

Not observed is not the same as did not occur.

And the central architectural principle is:

A fused conclusion should remain traceable to the observations that support it.

Why This Project Exists#

Multi-camera pattern-recognition systems rarely receive perfectly aligned, complete, equally reliable evidence.

Different cameras may disagree because of:

• viewpoint changes,
• modality differences,
• local clock offsets,
• occlusion,
• sensor outages,
• incomplete track overlap,
• appearance shift,
• different spatial coverage,
• different confidence scales,
• missing observations,
• contradictory local predictions.

A conventional fusion pipeline often tries to reduce all of this evidence to one final label, one identity, or one incident decision.

That compression creates a serious auditability problem.

If a system decides that two tracks belong to the same person, or that an event occurred across multiple cameras, a reviewer should be able to answer:

• Which cameras supported the conclusion?
• Which observations were used?
• Which observations were unavailable?
• Which alternative hypotheses were rejected?
• Why were they rejected?
• How confident was the system?
• Did two sources contradict each other?
• Was a missing camera silently interpreted as negative evidence?
• Can the final graph edge be traced back to its source evidence?

IncidentGraph is built around these questions.

Rather than making provenance a post-hoc explanation, the framework carries provenance through the entire evidence-fusion process.

Project at a Glance#

What Makes IncidentGraph Different#

Important

IncidentGraph is not presented as a real-world end-to-end surveillance system with solved cross-camera identity reconstruction. The current real experiment validates the local activity-evidence component, while the complete graph pipeline is separately exercised under controlled synthetic ground truth.

Reader Walkthrough#

A reader can understand IncidentGraph in six steps:

1. Evidence enters the system from multiple cameras or sensor modalities.
2. Each observation is normalized into a neutral evidence record containing camera, modality, local track, time, location, appearance, quality, and an immutable evidence reference.
3. Local recognition produces activity and event proposals without directly deciding the final incident graph.
4. Cross-camera association combines appearance, time, space, modality, and observation quality.
5. Event proposals are mapped onto fused entities, scored, deduplicated, and checked for contradictions.
6. The final graph retains selected events, provenance, rejected alternatives, confidence, missing-evidence objects, and an auditable timeline.

System Architecture#

IncidentGraph separates the system into four practical layers.

Three-Phase Reconstruction Workflow#

The separation is deliberate.

A stronger local activity model, re-identification model, detector, or tracking model can be inserted without changing the graph contract.

Heterogeneous Observation Record#

Conceptually, each observation is represented as:

where:

This neutral interface allows multiple evidence sources to enter the same graph layer.

A future implementation can therefore combine camera observations with:

• access logs,
• detector events,
• tracking output,
• sensor events,
• rule-engine results,
• human annotations,
• other structured evidence.

Typed Incident Graph#

The reconstruction output is represented conceptually as:

where:

• $V$ contains entity and location nodes,
• $E$ contains event relations,
• $C$ contains contradiction groups and alternatives,
• $M$ contains missing-evidence objects.

An accepted event edge stores:

• source entity,
• target entity,
• relation,
• confidence,
• time,
• provenance,
• alternative hypotheses.

Provenance Faithfulness#

A core evidentiary requirement is:

In practical terms:

A fused event should not exist without a pointer to supporting evidence.

This is one of the main differences between IncidentGraph and a pipeline that produces only an opaque final label.

Non-Negative Missingness#

If a sensor is unavailable over a time interval, IncidentGraph records a missing-evidence object rather than inventing a negative claim.

This distinction prevents the logical error:

The correct interpretation is:

Contradiction Preservation#

Two event proposals can disagree even when they refer to the same target relation and overlap in time.

IncidentGraph does not erase the losing hypothesis.

Instead:

1. competing proposals are grouped,
2. the highest-scoring claim can be marked selected,
3. the lower-scoring alternatives remain stored,
4. their subjects, scores, and provenance remain inspectable.

This makes the graph suitable for audit and human review.

Actor-Centric Activity Recognition#

The current real-data activity study converts MEVA event segments into:

• 16-frame clips,
• 112 × 112 spatial inputs,
• 28 activity classes.

In actor-centric mode, each clip is cropped around the annotated actor tube before resizing.

Claim boundary

These actor tubes come from annotations. Therefore, the actor-centric condition is an oracle-localization experiment. It isolates recognition after localization and should not be described as a complete detector-plus-recognizer pipeline.

The evaluated backbone is R3D-18, initialized from video pretraining and followed by a 28-class classification head.

Activity Training Setup#

Class-Imbalance Strategies#

Four principal training strategies were evaluated:

1. Unweighted training
2. Tempered inverse-frequency weighted loss
3. Weighted random sampling
4. Weighted loss + sampling

The experiment is intentionally reported across multiple seeds instead of selecting only the strongest run.

Cross-Camera Association#

The transparent reference association combines:

• appearance,
• corrected time,
• space,
• modality,
• observation quality.

The connected-component step is intentionally transparent.

A reviewer can inspect which pairwise association edge caused two observations to be merged.

Modality-Corrected Appearance#

Different modalities can introduce systematic shifts into an appearance embedding.

The controlled benchmark applies a modality-dependent correction before normalized cosine similarity.

This correction is deliberately modular.

A future deployment could replace it with:

• learned cross-modal alignment,
• RGB–thermal re-identification,
• domain adaptation,
• modality-specific encoders,
• cross-attention.

The observation and graph interfaces would remain unchanged.

Event Reliability#

A local proposal contains a model confidence, but confidence alone is not enough.

The controlled reference implementation also considers:

• source observation quality,
• number of evidence references,
• explicit conflict state.

The goal is to score an evidentiary proposal rather than blindly copy the local model’s softmax value into the final graph.

Duplicate Suppression#

Multiple local recognizers can generate duplicate proposals for the same physical event.

IncidentGraph creates an event signature using:

• relation,
• fused subject,
• fused object,
• temporal bucket.

The strongest duplicate is retained.

Unlike ordinary suppression that can discard context, the retained event still carries its evidence provenance.

Graph Reconstruction Algorithm#

1
Input:
2
    observations O
3
    event proposals R
4
    missing-evidence records M
5

6
1. Correct modality-dependent appearance and camera times.
7
2. Compare cross-camera observations of compatible entity types.
8
3. Compute association scores.
9
4. Add pairwise association edges above the relevant threshold.
10
5. Build cross-camera entity clusters as connected components.
11
6. Map event subjects and objects to fused entity clusters.
12
7. Compute event reliability.
13
8. Build the conflict graph.
14
9. Select the strongest claim in each conflict group.
15
10. Preserve losing claims as alternatives.
16
11. Suppress duplicate event proposals.
17
12. Create typed graph nodes and event edges.
18
13. Require non-empty provenance on accepted event edges.
19
14. Attach contradiction groups.
20
15. Attach missing-evidence objects.
21
16. Return the final incident graph and timeline.

Real MEVA Experimental Protocol#

The real activity-recognition component uses a bounded subset of official MEVA annotations.

The final split is video-disjoint.

Main Three-Seed Real-MEVA Results#

Mean ± sample standard deviation:

The strongest mean real-data result is the unweighted actor-centric fully fine-tuned family.

Why Multi-Seed Reporting Matters#

The combined loss-plus-sampler configuration contains the strongest individual run:

• seed 21 macro-F1: 0.1055

But across all three seeds the same family averages:

• macro-F1: 0.0624 ± 0.0373

This is a useful example of why reporting only the best random seed would overstate typical performance.

Actor Crop and Backbone Ablation#

A controlled seed-42 weighted-loss ablation isolates spatial focus and backbone adaptation.

Key observations:

• actor cropping improves macro-F1 from 0.0275 to 0.0471 under full fine-tuning,
• this is a 71.5% relative increase,
• full fine-tuning improves the actor-crop condition relative to a frozen backbone,
• scene context does not compensate for visual clutter in this bounded subset.

Calibration#

IncidentGraph reports probability quality in addition to classification performance.

The real activity study includes:

• raw Expected Calibration Error,
• temperature-scaled ECE,
• raw multiclass Brier score,
• calibrated Brier score.

Temperature scaling changes confidence values but does not change the predicted class.

This matters because a graph-reconstruction system should not treat an uncalibrated probability as automatically meaningful evidence strength.

Test-Set Dependence#

The test split contains 439 clips, but they come from only 12 source videos.

The concentration is substantial:

• largest source video: 175 / 439 clips = 39.9%
• largest two source videos: 260 / 439 clips = 59.2%

Therefore:

439 clips must not be interpreted as 439 independent incidents.

The reporting protocol uses source-video-aware sensitivity analysis rather than relying only on clip-level uncertainty intervals.

Statistical Reporting Protocol#

Three complementary summaries are used:

Failure Analysis#

IncidentGraph deliberately reports failure modes rather than only aggregate scores.

Important observations include:

• 9 of 28 classes have mean F1 equal to zero for every principal three-seed training strategy,
• 155 / 439 = 35.3% of test clips are never predicted correctly across the twelve principal runs,
• several balancing strategies are strongly seed-sensitive,
• class frequency still has a large relationship with recognition performance.

These limitations are part of the research result.

Controlled IncidentGraph System Diagnostic#

The complete graph pipeline is additionally exercised on IncidentGraph-Synth, a controlled software benchmark.

The benchmark contains:

• 60 generated incidents,
• RGB / thermal / depth modalities,
• missing-evidence rate: 0.12,
• contradiction rate: 0.35,
• generated ground-truth entity correspondence,
• generated event graphs,
• contradiction groups,
• missing-evidence annotations.

The purpose is to verify system semantics under known ground truth.

It is not presented as a real multi-camera publication benchmark.

Controlled Baselines#

Four methods share the same graph and evaluation interface:

Controlled Diagnostic Results#

Mean ± sample standard deviation across the 60 generated incidents:

IncidentGraph obtains the highest diagnostic graph score on:

• 45 / 60 incidents

Appearance-only is highest on:

• 15 / 60 incidents

The other two baselines are highest on none.

An Important Result: Best Entity F1 Is Not Best Graph Reconstruction#

Appearance-only produces slightly higher mean entity F1 than IncidentGraph in the controlled diagnostic.

That is useful rather than embarrassing.

It shows that:

Optimizing pairwise identity association is not identical to optimizing the complete evidentiary reconstruction.

IncidentGraph gives up a small amount of entity F1 while preserving:

• contradiction semantics,
• missing-evidence semantics,
• provenance,
• event relations,
• alternatives.

This is exactly what the graph representation was designed to evaluate.

What the Synthetic Diagnostic Does Not Prove#

Critical claim boundary

The controlled results verify software behavior under generated ground truth. They do not establish generalization to an unseen real camera network.

The diagnostic uses generated:

• modality bias,
• clock offsets,
• proposal quality,
• missingness,
• contradiction patterns.

These generated factors cannot substitute for independent real sensor shift.

A publication-level end-to-end claim still requires real experiments for:

• cross-camera identity association,
• temporal event linking,
• graph node/edge evaluation,
• real missing-camera behavior,
• real contradictory evidence,
• graph-level calibration,
• camera-network scaling.

Repository Structure#

1
IncidentGraph/
2
├── adapters/
3
│   ├── common.py
4
│   ├── cuva.py
5
│   ├── meva.py
6
│   ├── meva_kpf.py
7
│   ├── mtmmc.py
8
│   └── wildtrack.py
9
│
10
├── app/
11
│   ├── api.py
12
│   └── dashboard.py
13
│
14
├── configs/
15
│   ├── benchmark.yaml
16
│   ├── synthetic.yaml
17
│   └── real_meva/
18
│       └── default.yaml
19
│
20
├── docs/
21
│   ├── ARCHITECTURE.md
22
│   ├── BENCHMARK_PROTOCOL.md
23
│   ├── DATASETS.md
24
│   ├── ETHICS.md
25
│   ├── MEVID_ENTITY_ASSOCIATION.md
26
│   └── REAL_MEVA_AND_DEEP_TRAINING.md
27
│
28
├── incidentgraph/
29
│   ├── deep/
30
│   │   ├── activity_model.py
31
│   │   ├── metrics.py
32
│   │   ├── reid_dataset.py
33
│   │   ├── reid_model.py
34
│   │   └── video_dataset.py
35
│   ├── baselines.py
36
│   ├── benchmark.py
37
│   ├── calibration.py
38
│   ├── evidence.py
39
│   ├── fusion.py
40
│   ├── io.py
41
│   ├── metrics.py
42
│   ├── synthetic.py
43
│   └── types.py
44
│
45
├── scripts/
46
│   ├── benchmark.py
47
│   ├── build_meva_activity_manifest.py
48
│   ├── download_meva_annotations.py
49
│   ├── download_meva_annotated_subset.py
50
│   ├── evaluate_meva_activity.py
51
│   ├── evaluate_meva_activity_extended.py
52
│   ├── evaluate_mevid_reid.py
53
│   ├── generate_synthetic.py
54
│   ├── inventory_meva_annotations.py
55
│   ├── rebalance_meva_test_split.py
56
│   ├── run_real_meva_pipeline.ps1
57
│   ├── train_meva_activity.py
58
│   ├── train_meva_activity_resumable.py
59
│   └── train_mevid_reid.py
60
│
61
├── tests/
62
│   ├── test_api.py
63
│   ├── test_benchmark.py
64
│   ├── test_meva_kpf.py
65
│   └── test_synthetic.py
66
│
67
├── Dockerfile
68
├── docker-compose.yml
69
├── pyproject.toml
70
├── requirements.txt
71
├── requirements-deep.txt
72
└── README.md

Large checkpoints, raw datasets, videos, and generated experiment outputs are intentionally excluded from the source repository.

Important Project Paths#

Local Setup#

Create a Python environment:

1
python -m venv .venv
2
Set-ExecutionPolicy -Scope Process -ExecutionPolicy RemoteSigned
3
.\.venv\Scripts\Activate.ps1

Install the base dependencies:

1
python -m pip install --upgrade pip
2
pip install -r requirements.txt

For the deep-learning stack:

1
pip install -r requirements-deep.txt

Run Tests#

1
python -m pytest -q

The repository includes tests for:

• API behavior,
• benchmark behavior,
• MEVA KPF handling,
• synthetic graph behavior.

Inspect Training and Evaluation Commands#

1
python .\scripts\train_meva_activity.py --help
2
python .\scripts\train_meva_activity_resumable.py --help
3
python .\scripts\evaluate_meva_activity.py --help
4
python .\scripts\benchmark.py --help

Real MEVA Workflow#

Relevant scripts include:

1
scripts/download_meva_annotations.py
2
scripts/download_meva_annotated_subset.py
3
scripts/inventory_meva_annotations.py
4
scripts/build_meva_activity_manifest.py
5
scripts/rebalance_meva_test_split.py
6
scripts/train_meva_activity.py
7
scripts/train_meva_activity_resumable.py
8
scripts/evaluate_meva_activity.py
9
scripts/evaluate_meva_activity_extended.py

Controlled Graph Workflow#

Relevant components:

1
configs/synthetic.yaml
2
scripts/generate_synthetic.py
3
scripts/benchmark.py
4
incidentgraph/synthetic.py
5
incidentgraph/benchmark.py
6
incidentgraph/baselines.py

Datasets and Adapters#

The repository contains integration points for multiple multi-camera datasets.

Responsible Use#

Multi-camera fusion can affect privacy and civil liberties.

IncidentGraph is intended for lawfully acquired evidence with:

• appropriate access control,
• retention policies,
• human review,
• explicit uncertainty handling.

A graph edge should be interpreted as:

an evidentiary hypothesis

not automatically as:

• proof of guilt,
• proof of intention,
• a protected-attribute inference,
• a substitute for human judgment.

Low-confidence and contradictory hypotheses should be routed to review rather than converted directly into adverse decisions.

Research Boundaries#

The current project supports real-data statements about:

• local MEVA activity-evidence recognition,
• actor-crop versus full-frame behavior,
• frozen versus fully fine-tuned backbones,
• imbalance strategies,
• calibration,
• class-wise failure analysis,
• source-video dependence.

The current project does not yet support a real end-to-end superiority claim for:

• cross-camera identity fusion,
• complete real incident graph reconstruction,
• graph-level contradiction handling on independent real data,
• graph-level missing-evidence reasoning on independent real data,
• learned end-to-end graph fusion versus directly comparable SOTA systems.

Keeping this boundary explicit is part of the research design.

Strongest Next Experiment#

The highest-priority next study is a real end-to-end comparison using the same real camera evidence:

1
time-only
2
    ↓
3
appearance-only
4
    ↓
5
naive fusion
6
    ↓
7
fusion without provenance
8
    ↓
9
fusion without uncertainty / missingness
10
    ↓
11
complete IncidentGraph

The study should measure:

• cross-camera entity association,
• temporal event linking,
• graph node/edge quality,
• contradiction handling,
• missing-source handling,
• calibration,
• robustness to camera removal,
• scalability with cameras and tracks.

Future Architecture#

Potential extensions include:

• detector/tracker-derived actor tubes,
• learned RGB–thermal identity alignment,
• heterogeneous graph transformers,
• relation-specific message passing,
• graph-level calibration,
• camera-neighbor blocking,
• approximate-nearest-neighbor association,
• independent real multi-camera validation.

Reproducibility Checklist#

For every reported experiment, retain:

• exact Git commit,
• configuration file,
• dataset subset,
• split definition,
• random seed,
• checkpoint-selection criterion,
• environment/dependency versions,
• output directory,
• evaluation script.

Useful commands:

1
git rev-parse HEAD
2
git status
3
python --version
4
pip freeze

Project Links#

• Repository: github.com/dranubhaparashar/IncidentGraph
• Wiki: IncidentGraph Wiki
• Architecture: Architecture
• Real MEVA experiments: Real MEVA Experiments
• Controlled diagnostic: Synthetic Diagnostic
• Research boundaries: Research Boundaries
• Responsible use: Responsible Use

Summary#

IncidentGraph explores a different question from ordinary multi-camera classification:

How can heterogeneous evidence be fused without losing the information needed to audit the conclusion?

The current work combines:

• actor-centric video activity recognition,
• uncertainty and calibration analysis,
• cross-camera association,
• explicit provenance,
• contradiction preservation,
• missing-evidence reasoning,
• typed incident graphs,
• reproducible real-data component evaluation,
• controlled full-system diagnostics.

The project does not claim that uncertainty has disappeared.

Instead, it aims to ensure that uncertainty, disagreement, provenance, and missing observations remain visible in the reconstructed incident graph.