Other agents are part
of the environment.
Mule is an early-stage effort to build reinforcement learning environments for social games. We’re interested in how agents learn to act when information is incomplete, communication is strategic, and other players are adapting too.
The setting
We frame these games as partially observable stochastic games. Each agent receives a different observation of the underlying state, maintains a history of messages and events, and selects both language actions and game actions. Rewards can be individual, shared within a team, or opposed across teams.
hᵢ,ₜ = (oᵢ,₀, aᵢ,₀, …, oᵢ,ₜ)
The difficult part is not just inferring a hidden role. It is reasoning about what another agent has observed, what it believes, and how its next action depends on the conversation so far.
Questions we want to study
- Belief tracking. Can agents update calibrated beliefs about hidden states as new evidence arrives? Can they distinguish a player’s claim from a reliable observation?
- Communication and coordination. When does dialogue improve joint decisions? How do agents coordinate with unfamiliar partners when their information differs?
- Long-horizon credit assignment. Which messages and decisions contribute to an eventual outcome? How do terminal rewards compare with intermediate signals tied to inference or coordination?
- Population generalization. Do learned policies transfer to new opponents, roles, rules, and communication styles, or merely exploit a familiar training population?
Evaluation direction
Our intended approach pairs game outcomes with behavioral measurements. Win rates alone can conceal role imbalance, weak opponents, or brittle conventions. Useful evaluations should vary player populations and random seeds, separate training from evaluation opponents, and report uncertainty alongside aggregate performance.
We are also interested in trajectory-level analysis: what information was available at each turn, how beliefs changed, and whether an agent’s stated plan matched its subsequent actions. These are research priorities, not claims of completed benchmarks or demonstrated transfer beyond games.
Related research
Work informing the direction. These are independent projects, not Mule results.
- The benchmark where AIs have to lie to win ↗
A direct inspiration: model-vs-model social deduction makes sustained strategic behavior and coordination visible in full game trajectories.
- CICERO: language and strategic reasoning in Diplomacy ↗
Combines language generation with planning and reinforcement learning for negotiation and coordination. A concrete example of coupling dialogue to strategic objectives.
- Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game ↗
Uses an LLM to reason and propose candidate actions, then an RL policy to select among them. Relevant to separating fluent communication from effective decision-making.
- Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot ↗
Evaluates generalization to unfamiliar social situations and partners. Its central lesson for us: training-population performance is not enough.
- Playing social deduction games with reinforcement fine-tuned large language models ↗
Reports limitations of directly optimizing terminal win–loss rewards, and studies more targeted signals for social reading and influence. Relevant to reward design in hidden-role games.
Get in touch
We’d like to hear from people working on multi-agent RL, social reasoning, and game-based evaluation.