Better Trivia
Summary
The problem. A trivia night is won or lost on whether people think the questions were fair. The arguments that spoil a night follow a handful of patterns we know well: a question with two defensible answers, a name that people know by a nickname, an opinion dressed up as a fact, an answer that is given away by the question itself. When one turns up in a room, the host has to settle it on the spot, and the winner is whoever argued best.
What we do about it.
- A written standard. Every question has one defensible answer and states its own standard (“at birth”, not just “first name”).
- A checker that reads every quiz first. Rules catch what a machine can know for certain and say it in the creator's own words, with a fix. When a rule changes, every quiz is checked again. The checker also plays each quiz start to finish with synthetic players on the real game engine; the whole library of 147 quizzes takes about 30 seconds.
- No paper to argue over. Answers are scored by the server on timed phone screens, so the debates about scoring never start.
What we found when we pointed it at ourselves. The four newest rules found no new Problem or Warning in the 146 real quizzes of the library (three Tips). Our first draft of them did: it treated two different people with similar names as one name, and a Tip fired 63 times. Running the rules over real quizzes is how we caught both, before anything shipped.
How serious is a finding?
A Problem breaks or spoils a game.
It is the checker's highest alarm. A quiz with a Problem does not count as known good until it is fixed, and the checker looks at it again every time a rule changes.
The rules behind the demo
| Rule | Severity | What it catches | Demo |
|---|---|---|---|
several-correct | Problem | Arithmetic choices that really come out the same | Q1 |
opinion-question | Problem | “My favorite…”, “do you prefer…”: nothing can be wrong | Q3 |
nickname-choices | Warning | A name and its nickname both offered | Q5 |
name-standard-unstated | Tip | A first or real name asked for without saying which counts | Q5, Q7 |
similar-choices | Warning | One answer is the other with extra words | Q9 |
catch-all-choice | Warning | “All of the above” | Q10 |
leak-own-answer | Problem | The question names its own answer | Q11 |
| Passes clean | The repaired version of each trap, to show the rule does not cry wolf | Q2, Q4, Q6, Q8 | |
The checker has many more rules (placeholder text, too little time to read, broken pictures and others); the demo covers the ones about arguments.
Questions people ask
Can software really tell whether a question is ambiguous?
Some of it, for certain: arithmetic can be worked out, opinion wording can be recognised, and a name and its nickname can be spotted. Two true facts (“the largest state” by area or by population) cannot, so that stays with the quiz's author, and the checker says plainly what it cannot see.
What happens when the checker flags a quiz?
The creator sees the finding in plain words with something to do about it. A Problem means the quiz does not count as known good until it is fixed. Nothing is locked automatically: the decision to use a quiz stays with a person.
How do we know the checker itself is right?
The audit quiz in the panel is the test. Each trap must raise exactly the finding it was built for and each control must pass clean. It is run against the real library too, which is how the false alarms in our first draft were found and fixed.