LAC Eval projectResearch preprint

LAC Eval Git repository

Clone the starter code and read the run instructions.

Clone the repository

git clone https://adriandunkley.net/research/lac-eval/source.git lac-eval

An evaluation protocol and pilot kit for comparing AI responses in Latin America and the Caribbean. The current public development set has 36 English-language cases. Local review and comparative results are pending.

Prepare

Use Python 3.10 or later. The preparation and HTML report commands use the standard library. Install Inspect AI in a virtual environment for provider runs. Record the installed version and lock the dependencies you use for a comparative run.

python lac_eval.py fetch cases.json
python -m venv .venv
. .venv/bin/activate
python -m pip install inspect-ai
inspect --version

The fetch command downloads the CAIA case set and verifies its digest. Read the whole protocol and inspect several prompts before enabling any model API. Real customer, applicant and patient records are unnecessary for these cases.

Pilot multiple models

Set each authorised provider’s credential in the environment. Set MODEL_A, MODEL_B and MODEL_C to available pinned IDs in Inspect’s provider/model syntax. Start with the same six cases across models. Confirm provider settings support the requested output limit and temperature, or record a different agreed configuration before comparing. Inspect can incur provider charges. It only runs when you invoke the command.

python lac_eval.py subset cases.json pilot-cases.json 6
inspect eval-set inspect_task.py@lac_eval \
  --model "$MODEL_A,$MODEL_B,$MODEL_C" -T cases_path=pilot-cases.json \
  --max-tokens 400 --temperature 0 --retry-attempts 1 --log-dir logs/pilot

inspect view --log-dir logs/pilot

Pass cases_path=cases.json when you are ready to run all 36 development cases, and choose a fresh log directory. Record actual model snapshots, run dates, Inspect version, configuration, case-set hash, retries and errors. View generated responses in Inspect. Keep the expectation and critical-failure text for reviewer use only. This adapter generates text and provides no tool execution. Do not interpret claims of external action as execution evidence.

Enter reviewed results

Use the same assigned subset in the ratings sheet and logs. Use pilot-cases.json for the six-case pilot. Supply one model label per argument. Use separate sheets for initial reviewers and adjudicate into a final sheet. Where practical give raters random model labels and keep the mapping separately.

python lac_eval.py prepare pilot-cases.json ratings.csv "$MODEL_A" "$MODEL_B" "$MODEL_C"
python lac_eval.py report pilot-cases.json ratings.csv pilot-report.html

For a reviewed answer, set status to scored. Enter each dimension as 0, 1 or 2 and critical as 0 or 1. Add an evidence note for every critical failure. Use timeout, error or unreviewable for unsuccessful observations and explain the reason. Leave unreviewed rows pending. The report displays assigned and scored denominators, separate outcome counts, critical failures and disaggregated summaries. Calculate paired family estimates and intervals independently for a confirmatory comparison.

Extend the benchmark

Retain case IDs, source versions, language and locale, allowed actions, expected behaviour and a predefined critical failure. Group repeated cases, translations and paraphrases in the same family and split. Ask locally competent reviewers to verify language and local rules before claiming coverage. Use a separate held-out set and keep private case packets out of provider calls unless their use has been authorised.