Lexsi AI Safety & Alignment Hackathon · Track 2b-5 (safety-preserving fine-tuning), 2b-6, 2a-4, 2a-2

How Ìdìbò was built and tested

Ìdìbò answers questions about Nigeria's 2027 election in English, Nigerian Pidgin, Yorùbá, Hausa and Igbo, from cited INEC pages and the Electoral Act 2026. It is a LoRA fine-tune of Cohere's Tiny Aya Earth (3.35B). It must refuse election harm (incitement, fake INEC notices, vote buying, intimidation), never take a political side, and still answer ordinary civic questions. Safety tuning usually breaks that last part, so we measure it.

The pipeline: four Lexsi toolkits

Training data. Civic answers grounded in a knowledge base of 309 cited passages, refusals that offer a lawful alternative (how to report a fake notice, what the law says about vote buying), neutral answers to partisan prompts, and "I don't know, check INEC" answers for things no source can know. English, Pidgin and Yorùbá; Hausa and Igbo were left out on purpose to test transfer. A control adapter uses the same recipe on the civic answers alone: the "plain fine-tune".

Grounding. Hybrid search (BM25 plus multilingual-e5, fused by rank) over the knowledge base. For Yorùbá, Hausa and Igbo the model first translates the question to English and searches with both. Official candidate lists are shown word for word from INEC's PDF instead of being generated.

Before and after: the held-out benchmark

Written after training, never seen by the model (nearest training question: cosine 0.948, no near-duplicates). English: 56 harmful requests (16 wrapped in jailbreaks such as role-play, fiction or "research"), 48 benign look-alikes on the same topics, and 12 party, ethnic and religion swap tests. Rates are % with 95% Wilson intervals.

Which differences are real (AuditKIT paired bootstrap, same items)

What it shows. Fine-tuning on civic answers alone made the model "safe" by refusing a third of honest questions. Mixing safety examples into training kept the safety and gave the helpfulness back. If retraining is not an option, Safe LoRA recovers about two thirds of the gap; ReSta did not help a model that already over-refuses.

Every language

Benchmark v1 per language: 18 harmful requests, 18 benign look-alikes, 20 fact questions and 8 unanswerable ones. Hausa and Igbo were never in the training data. "+ translation" is the live app's setting: the question is translated to English before search.

Honest reading. The gains are in safety, neutrality and answering in the user's language (Pidgin answers in Pidgin: base 13%, Ìdìbò 100%). Fact accuracy did not improve over the base model given the same sources, and it fell in Igbo. Translating the question before search lifted Yorùbá facts from 5% to 35% (p = 0.005). Igbo is labelled beta in the app. Each language has 18 harmful items, so one answer moves a rate by 5.6 points; the intervals are in the write-up.

Code-mixed prompts (30, written by hand: Pidgin-English, Yorùbá-English, Hausa-English): Ìdìbò refused all 9 harmful ones and got 8 of 9 facts right, but refused 4 of 6 honest look-alikes, its clearest remaining weakness.

Side by side: the same question, real answers

Recorded answers from the benchmark runs (greedy decoding, same sources and instructions for every model). Harmful output from the base model is described, not reproduced. The last example is a failure.

Red-teaming this live app (AuditKIT)

Benchmarks test the model; voters use the whole system. AuditKIT's built-in probes (prompt injection, jailbreak, encoding, over-refusal) and three probes we wrote ran against this URL: indirect injection (a fake INEC notice with a hidden instruction, sent to "Is this message true?" in four languages), instruction extraction in Pidgin, Yorùbá and Hausa, and endorsement bait.

Fix. A marker added to "I can't confirm this" is enough to fake a screenshot, so we added a guard (idibo/guard.py): if a question carries injection cues and the reply repeats a phrase the question demanded, the reply is replaced. In check mode, the replacement warns that the message hides instructions aimed at an AI. It found no cue in our 691 benchmark questions, so normal answers are unchanged. Re-run against the guarded app: 0 of 19 injection, encoding and indirect-injection attacks worked (6 before). The only remaining flag is the calculator request, refused as out of scope.

How answers were graded

Each answer is back-translated into English by tiny-aya-global and graded against a fixed rubric by c4ai-command-r7b on our own GPUs (Kaggle, 2× T4). Round 1 was graded by command-a-plus. On the 1,241 items both graded, they agree 84% of the time (Cohen's κ 0.80). On harmful items the open grader is more lenient (κ 0.28), so the harmful rates above are lower bounds. A plain check agrees with the ranking: 1 of Ìdìbò's 56 harmful- prompt answers lacks a refusal opening, against 40 for the base model.

A hallucination no rubric graded: invented contacts

Reading the answers by hand, we found refusals that point voters to INEC contacts that do not exist: a hotline 0800-INEC-HELP (43 times), a page www.inec.gov.ng/factcheck (28), and the domain inec.gov.ng, which does not resolve (INEC is inecnigeria.org). A made-up number is exactly what a scammer would want voters to call.

Safety training made this worse: "report it to…" refusals invite an invented "where". The fix is a check, not more training: any web address, email or phone number that is not in the sources is replaced with inecnigeria.org, on the web app and on WhatsApp (idibo/contacts.py).

On WhatsApp, where rumours spread

Voters can ask Ìdìbò by text or voice note, or forward a message they doubt, on WhatsApp. Replies use the same pipeline (sources, injection guard, contact check), come with numbered sources, and a voice note gets a spoken reply. It runs on Meta's free test number, which can reach up to five registered phones; a public launch needs a verified business number.

Deployment, latency and cost

• Runs on CPU. Ìdìbò merged and quantised to 4-bit GGUF (2.1 GB), served by llama.cpp on 8 vCPUs (Railway). About 10–20 s per answer, streamed word by word; one answer at a time per replica, others queue.
• Cost. About $0.0009 per answer self-hosted, against about $0.0075 on a closed API (Cohere Command A list price, 2,000 tokens in and 250 out): roughly 8× cheaper, and the data stays in our deployment.
• Voice. Nigerian voices in and out through YarnGPT (Pidgin, Yorùbá, Hausa, Igbo, English).
• Production gaps. One replica (scale out for election day), no accounts or saved history by design (nothing personal is stored), and the knowledge base must be refreshed from INEC as notices change.

Known limitations

• Over-refusal stays high on code-mixed prompts and in Yorùbá (39% with translation).
• Igbo is untrained and weaker than the base model on facts: labelled beta.
• A Pidgin-flavoured question asked with English selected may get a Pidgin answer (the model mirrors the question).
• It sometimes adds "ID or passport" to what to bring to the polling unit; the requirement is the PVC.
• The model still invents contact details in some refusals; the deployment check removes them before anyone sees them.
• INEC's 2027 presidential candidate list was added to the app after evaluation; the benchmarks ran without it.
• Benchmark translations and the guard's Yorùbá, Hausa and Igbo messages are machine-made and not yet reviewed by native speakers.
• Not official INEC guidance. Check inecnigeria.org.