Institutional Agent Audit — Anonymous Release

This repository accompanies the EMNLP 2026 Industry Track submission "Discovering Institutional Biases in LLM Agents Through Emergent Process Simulation". It contains the audit framework code, experimental data, and process traces needed to reproduce the paper's analyses. The ground-truth dataset will be released upon acceptance.

Contents

.
├── README.md
├── code/
│   ├── generate_figures.py    # Generate all 3 paper figures from data/
│   ├── extract_data.py         # Parse raw experiment artifacts → CSVs
│   └── tier3_experiments.py    # Sycophancy control + vanilla LLM baseline
├── data/
│   ├── country_summary.csv     # Per-country × condition × round stance summary
│   └── process_shifts.csv      # Per-agent stance-change tracking
├── experiment_configs/
│   ├── exp2_govt_only_init_config.json  # Agent profiles + env for Govt-Only
│   ├── exp4_open_init_config.json       # Agent profiles + env for Open
│   └── example_steps.yaml               # Full deliberation workflow (1 round)
└── process_traces/
    ├── sample_deliberation_traces.json   # 12 annotated deliberation transcripts
    └── per_country_shift_summary.json    # Per-country shift rates (Govt-Only vs Open)

Quick Start

Reproduce figures

cd code
python3 generate_figures.py
# Output: fig1_gradient.pdf, fig2_process.pdf, fig3_crossmodel.pdf

The figures are generated from data/country_summary.csv and data/process_shifts.csv, which contain the aggregated results of all experiments described in the paper.

Reproduce data extraction

If you have raw experiment artifacts (questionnaire JSON files from AgentSociety2 runs), place them in an artifacts/ directory and run:

python3 extract_data.py
# Output: data/all_results.csv, data/country_summary.csv, data/process_shifts.csv

Run Tier 3 experiments (sycophancy control + vanilla LLM baseline)

Requires an LLM API endpoint with the AgentSociety2 environment variables set:

export AGENTSOCIETY_LLM_API_KEY=your_key
export AGENTSOCIETY_LLM_API_BASE=your_base_url
export AGENTSOCIETY_LLM_MODEL=qwen3.6-35b-a3b

python3 tier3_experiments.py prepare   # Generate prompts
python3 tier3_experiments.py run       # Execute experiments

Data Format

country_summary.csv

Column	Description
country	Country name
condition	Govt-Only, Mentioned, Open, or Polling
model	LLM family (Llama 3.1-8B, Qwen 3.6-35B, GPT-4o)
tariff_round	1–5
delib_rounds	Number of deliberation rounds (max 3)
n_agents	Number of government agents (5)
wait_pct	% agents choosing Wait (A)
negotiate_pct	% agents choosing Negotiate (B)
retaliate_pct	% agents choosing Retaliate (C)

process_shifts.csv

Column	Description
country, condition, model, tariff_round	Same as above
agent_id	Unique agent identifier
n_delib_rounds	Rounds this agent participated in
initial_stance	A/B/C at deliberation round 1
final_stance	A/B/C at final deliberation round
n_shifts	Number of stance changes
shifted	True if stance changed at least once
shift_path	Full stance trajectory (e.g., "B→C→B")

Experiment Design

The paper tests four experimental conditions manipulating citizen-government information flow:

1. Govt-Only: No mention of citizens. Baseline.
2. Mentioned: Citizens mentioned but views withheld.
3. Open (primary): Citizen opinion distributions provided as voluntary messages.
4. Polling: Same data framed as scientific polling (±3% margin of error).

Each condition is tested across 28 countries, 5 tariff rounds, and 3–4 LLM families. Government agents (5 per country, with real institutional roles) deliberate for up to 3 rounds. See experiment_configs/ for the full agent profiles and deliberation workflow.

Reproducibility Notes

• All models have verified training cutoffs predating the February 2025 tariff events.
• Citizen opinion distributions were collected from independent surveys of 250 demographically-stratified citizen agents per country and held constant across deliberation models.
• Ground truth responses were compiled from authoritative trade policy trackers (C.H. Robinson, Blakes, PIIE, Trade Compliance Hub). See the paper Appendix for full source documentation.
• Three-run variance testing on Qwen 3.6-35B confirms primary findings are robust to run-to-run variation (see Appendix I in the paper).

Citation

If you use this code or data, please cite the accompanying paper. The full citation will be provided upon acceptance.

License

Code: MIT License. Data: CC-BY 4.0 (upon acceptance; ground-truth data release pending).