Polis, policy conversations the Wiki Way

Designing a deeper enagagement

Two companion posts on innovating with Wikimedia decision-making is underway, which makes the case to Wikimedians. This post is for the research side of my world: the people working in democratic innovation. It's about an alternative view on Polis — a tool designed for one-shot civic consultations with the anonymous public — increasing ambition to focus on deeper engagement. The result is wiki-polis, currently in beta at wiki-polis.toolforge.org (source on GitHub).

Wiki-polis design: participants can contribute in three phases. First, to find the best version of the statements. Second, to identify the best arguments for the most important statements. And finally, they can express their opinions on these statements, with the context of relevant arguments.

If you work in democratic innovation, you probably know Polis: participants vote agree/disagree/pass on short statements, submit statements of their own, and clustering reveals the opinion groups — and, more interestingly, the statements that bridge them. It is well-known because of the success stories in Taiwan and other places.

Almost all of those deployments share a shape: a convening institution runs a one-time consultation, with mostly anonymous participants who will never interact again, feeding into a decision that someone else will make. The tool is usually advertised as allowing participants to not only vote, but to propose new statements, allowing a sort of dialogue.

When I was looking for a tool that could help bring new life in Wikipedia's bottom-up policy making processes, Polis was the first software that came to mind. Unfortunately, when I was talking with (a very small sample of) policy makers that used it, it seemed that while the number of residents that participated was high, the number of people proposing alternative statements was fairly low, and the number of returnees was very low.

For Wikipedia, where the status quo is more akin to Request for Comments in most communities with perhaps too much interaction, I needed something that was closer to the spirit of a community where editing a page is the norm. Eventually, I managed to combine the Particiapi package with Polis and a lot of "vibe coding" to build a first prototype where there is a more explicit invitation to improve statements. Once I started this process and talked with my Wikipedian colleagues, I realized that merely producing a cluster map would likely not be enough. I see a lot of potential in Polis for this, but it needs some work to make it fit the promise.

Wikipedia communities self-govern: while there is often an explicit policy that says "Wikipedia is no Democracy", it may be one of the few top websites that gets even close. The community is both the consulted and the decider, and the policies they make bind their own behavior. And they deliberate constantly, and perhaps too much so. Unfortunately, those are often not very effective, and take a lot of energy. Across the largest Wikipedias, most rules are created in a community's early years and rule-making activity then falls off (Hwang & Shaw 2022); Wikipedia's formal mechanisms for norm articulation have measurably "calcified against changes — especially changes proposed by newer editors" (Halfaker et al. 2013); and a substantial share of English Wikipedia's RfCs never reach a conclusive close (Im et al. 2018).

So the premise of wiki-polis is that Polis solves real problems for this population, but not unmodified. Below are the changes, and the reasoning behind each. The guiding principle throughout: own the wrapper, not the engine. We run stock Polis as the clustering and voting engine, unforked, and build everything else around it.

0. Authenticated pseudonymity, with opt-in reveal

The first divergence that any user will notice, is that you need to authenticate, and choose a pseudonym. Wikipedians typically have their deliberations out in the open, for the world to read. Not that I would recommend it - it's pretty boring.

It would be a non-starter to keep the raw contributions to the process from Wikipedians. In fact, the entire attractiveness of Polis would be that people can look up how they compare to the broader field of opinions in a few clicks. I can imagine that there will be a number of Wikipedians who will want their vote to be connected to their on-wiki identity. At the same time, the fact that there is no direct exchange is a core benefit of Polis. After all, it is effects such as herding and anchoring that we want to avoid.

I found a compromise between allowing that, and encouraging anonymous participation: inside a conversation, participants vote under a per-conversation pseudonym — unique platform-wide and never reused, so pseudonyms can't be linked across conversations. While the conversation runs, nobody sees who voted what; opinions surface only as aggregate clusters. Once the process is over, and everyone can have a complete appreciation of what their contribution looks like, they have a limited timeframe to formally and publicly connect their on-wiki identity with their pseudonym. For everyone who doesn't, the internal account-to-pseudonym link is deleted after a retention window.

1. The question you can edit

Vanilla Polis allows participants to submit statements. It's not very active, but it's there. These participant statements are to be voted on when the user runs out of organizer statements.

In the Wikipedia context, the organizer is often likely not much more knowledgeable than the average participant about the topic. Why would we want to assume that the seed statements are anything close to a complete set? This seems entirely in line with the equality assumption that underpins the initial designs of Polis, and the importance it assigns to high quality statements (Small et al. 2021). There are two situations I could imagine that a participant should contribute an alternative: when they have a better phrasing of a statement that they are currently voting on, or when they see a gap that requires a statement. We implement this as a permanent invitation to 'improve the phrasing of a statement' and a limited invitation that only opens after 10 votes, to add a novel statement to the set (at most 3).

How this plays out in practice, we will have to see with real people, discussing real topics. I'm confident we will have to up our game in moderation, but this is something where iterations will do a better job than a fully developed game-plan.

Vote yes or no.
Suggest improved statements, or add missing statements

Contribute arguments on both sides per proposal
Prioritize the most helpful arguments

2. An argument layer

While regular Polis tells you what a community thinks, it does not tell you why. For a one-shot consultation feeding a government process with much more nuance and steps, that may suffice. The standard Polis output also requires substantial human analysis before it's consumable — Huang et al. (2023) note the automated report isn't usable without extensive manual work, which is part of what motivates their LLM experiments. In Wikipedia, I don't think we could get away with that. It would feel too much like a black box answer, and while the dataset is interesting, it does not help people gain insight to the point that they can determine their stance on the final product.

So wiki-polis adds a second phase: After the opinion mapping stabilizes, the organizer curates a small set of featured statements — I imagine 8–12, suggested by the system from the cluster analysis. This clustering is something that we will have to revisit as well, because I suspect we will have many more 'similar' statements because of our adjusted submission process. The goal will be to pick one statement for each cluster of similar statements that anchor a cluster of participants. I imagine that will be 8-15 statements in total: "featured statements". On these, and only these, an argument layer opens: participants write short pro and con arguments and select the most convincing argument to them. Arguments are atomic like statements, directional by design, and explicitly not inputs to the clustering. They're annotations for human readers, kept in our own database alongside the Polis vote data.

This is still in line with the fundamental design principle of Polis that drew us there: no threading, no responses. No attribution.

3. An informed opinion: asynchronous deliberative polling

The final phase will be familiar to anyone who knows Deliberative Polling (Fishkin 2009): measure opinion, expose people to balanced arguments, measure again. This does not claim to do the same as a weekend-long deliberation with a balanced information package and a panel of experts, but now that we have these arguments, why not repeat the earlier voting phase?

This time with only a core set of statements and no ability to add new nuance into the process, we invite participants one last time to share their insights. Technically, this is a second, independent Polis conversation — with the top-ranked pro and con arguments displayed alongside each statement. Read first, then vote; pass remains available and is meaningful ("I've considered both sides and won't commit").

Because both rounds rely in part on the same statements and the same pseudonymous participants, we might be able to ask a question that our current wiki processes — and most Polis deployments — cannot: did exposure to the community's best arguments actually change minds, and along which fault lines? The small fixed statement set also lowers the entry barrier, so participants who found the full voting loop too demanding can join only here.

This is, as far as I know, the main genuinely novel contribution: deliberative polling's before/after measurement, without the convening weekend, embedded in a tool a volunteer community can run itself.

Express opinions on the final set of statements, with their most helpful arguments

Composable phases

The full set forms a process with 7 phases where the organizer and the participants arternatingly contribute:

Seed the initial statements (organizer)
Collect initial opinions and refine the statements
Reflection on the cluster mapping and select representative statements (organizer)
Argument mapping
Clean up the final set of proposals and arguments. Prepare final participation round (organizer)
Informed vote
Prepare the report and data (organizer)

But the attentive reader will quickly realize that in reality, these are building blocks that don't have to be used as a complete series. When the organizer stops after the opinion mapping, you have a rich opinion map. Already more valuable than most on-wiki straw polls. If you stop after the arguments, you have a richer version even, and can create a great report that can help you even gain insight into which participants came up with arguments for their opponents' viewpoints. But if you want to consult a less edit-minded population, you could use your seed statements as input for the argument mapping, with or without the informed vote.

At the end of the process, we still don't have a balanced policy - but we do have the input we need to draft a compromise that can count on broad support. Where it differs from many straw polls in the traditional sense, is that it will allow us, after some work perhaps, to also investigate beyond what proposals can count on just enough support to pass, but to actually look at the proposals through the lens of equity: justified representation. What proposal could make most people at least somewhat satisfied with the outcome. This is far from straightforward, and would require some serious research to figure out.

The output feeds the community's normal consensus process, where a final RfC or vote — now anchored on mapped agreement and ranked arguments rather than a blank page — may actually work. Wikipedia's "not a democracy" norm is real, and a tool that tries to make decisions would be rejected; a tool that makes the eventual decision better-informed might not be.

Now in beta

Wiki-polis is in beta: stable enough to run consultations, still very much a prototype, GPL-licensed, running on Wikimedia's Toolforge. If you build deliberation tools, I'd genuinely like your criticism of these design choices — especially the gating and the identity model, where I expect the strongest disagreement. And if you've solved adjacent problems — statement routing, argument quality, cross-deployment comparison — I'd like to steal from you, openly, the wiki way.

As we engage in pilots with some community groups, we will finetune based on what we learn, and maybe make some larger decisions. For now, the focus is on building something that works well for the Wikimedia communities; hopefully we can make it useful enough that it will also see use outside! Once we see our first reports come out, I'll share more.

Acknowledgements

This prototype has been developed in collaboration with Christophe Henner and Alek Tarkowski from the Schrödinger Moonshot Collective, and received much valuable feedback from friends and colleagues across the Wikimedia movement. The software builds upon open source software from the Computational Democracy Project (GitHub) and a front-end API Particiapp by Polis-NL, CodeforNL (GitLab).

Page updated

Google Sites

Report abuse