Vision#

Large language models can be useful in in-vehicle AI systems, but deployment is difficult when memory and compute budgets are tight. This project explores a practical approach: combine 4-bit quantization with low-rank residual compensation so the model remains lightweight while recovering part of the quality lost through aggressive compression.

Why This Matters#

NOTE

The repository is explicitly framed as a minimal reproducible starter for an INT4 + low-rank adapter compensation pipeline.

IMPORTANT

The README describes an end-to-end flow: corpus → export → quantize (INT4) → eigenspace (SVD) → build adapters (U, D) → inject → inference → evaluation.

TIP

The starter is designed to prove the flow on a toy transformer first, then be swapped with a LLaMA-derived model later.

WARNING

The current public repo is a starter implementation, not a full production deployment kit.

CAUTION

Claims about real-world automotive performance should be validated with a real model, real deployment stack, and benchmark data.

What This Starter Does#

1
Corpus Preparation
2
        ↓
3
ONNX Export
4
        ↓
5
INT4 Quantization
6
        ↓
7
Eigenspace Computation (SVD)
8
        ↓
9
Build Low-Rank Adapters
10
        ↓
11
Inject Adapters
12
        ↓
13
Run Inference
14
        ↓
15
Evaluate Output

The repository README describes this exact flow and positions it as a proof-of-pipeline for memory-efficient inference.

Project Attributes#

Repository Structure#

The repository currently includes dedicated folders for configs, data, Docker, docs, environment setup, scripts, templates, tests, and source code under src/dac_q4_its.

1
Vehicle-Scale-LLMs-Integrating-Low-Rank-Residuals-and-4-Bit-Quantization-for-In-Vehicle-AI/
2
├── configs/
3
├── data/
4
├── docker/
5
├── docs/
6
├── env/
7
├── guidelines/
8
├── scripts/
9
├── src/
10
│   └── dac_q4_its/
11
├── templates/
12
├── tests/
13
├── Makefile
14
├── README.md
15
└── pyproject.toml

Quickstart#

The README provides this starter flow:

1
python -m venv .venv && source .venv/bin/activate
2
pip install -r env/requirements.txt
3
python scripts/01_prepare_calibration.py
4
python scripts/02_export_onnx.py
5
python scripts/03_quantize_int4.py
6
python scripts/04_compute_eigenspaces.py
7
python scripts/05_inject_adapters.py
8
python scripts/07_run_inference.py --prompt "avoid tolls and reach airport by 6pm"
9
python scripts/08_eval_nln.py

This sequence shows the intended end-to-end execution path from calibration preparation to inference and evaluation.

What the Starter Gives#

According to the README, the starter includes:

• Working INT4 quantization with per-row scales
• Rank-8 adapters computed from calibration activations
• A compressed forward pass based on dequantized INT4 matmul plus low-rank residual correction
• Simple EM / SOFT-F1 metrics demo

Core Compression Idea#

1
def compressed_forward(x, w_int4, scales, U, D):
2
    base = dequantized_int4_matmul(x, w_int4, scales)
3
    correction = U @ (D @ x)
4
    return base + correction

The exact README wording summarizes the compressed forward path as:

1
dequantized(INT4 matmul) + U(Dx)

That is the central systems idea of this starter.

Why This Design Is Interesting#

• It targets memory efficiency directly through INT4 quantization
• It adds a low-rank correction path instead of fully undoing compression
• It keeps the pipeline modular, with separate scripts for export, quantization, eigenspace computation, adapter injection, inference, and evaluation
• It is positioned as a bridge from a toy transformer to a real LLaMA-derived deployment path

Real-Model Upgrade Path#

The README suggests the following path to move beyond the toy transformer:

• Replace src/dac_q4_its/modeling/loader.py to load real HF / LLaMA weights
• Populate scripts/02_export_onnx.py to export real projections
• Replace toy heads with real Q / K / V / FFN matrices
• Keep the rest of the pipeline structure intact

This is useful because it shows that the starter is intentionally modular rather than being locked to the toy model.

Current Limitations#

• Public repo is still a starter, not a fully benchmarked release
• README is concise and focused on flow, not exhaustive evaluation
• Real deployment details for Jetson / TensorRT are not yet fully wired in the public starter
• The repo currently proves pipeline structure more than production-grade validation

What I Would Add Next#

• Benchmark comparisons against FP16 / INT8 / plain INT4 baselines
• Memory usage charts for embedded automotive targets
• End-to-end latency numbers on Jetson-class hardware
• Accuracy trade-off analysis as rank changes
• A worked example using a real LLaMA-derived checkpoint

These additions would make the repo stronger as both a research artifact and an engineering deployment guide.

Key Innovation#

IMPORTANT

The main innovation is the combination of INT4 quantization with a low-rank residual compensation path in a pipeline designed for constrained, vehicle-oriented inference settings.

Conclusion#

This repository is a strong starter for exploring vehicle-scale LLM inference under tight memory budgets. Its value is not only in compression, but in the fact that it demonstrates a full reproducible flow: prepare, export, quantize, compensate, inject, infer, and evaluate.

From full-precision overhead to vehicle-ready efficient inference — that is the direction this project is pushing toward.