# :smiling_imp: The MACHIAVELLI Benchmark

MACHIAVELLI is an RL-style gym environment for text agents. Our environment is based on human-written, text-based Choose-Your-Own-Adventure games containing over half a million scenes with millions of annotations. The games spotlight high-level social decisions alongside real-world goals to achieve, measuring an agent's ability to plan and navigate complex trade-offs in realistic social environments.

For more information, see our [website](https://aypan17.github.io/machiavelli/) and our paper: [Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the *Machiavelli* Benchmark.](https://arxiv.org/abs/2304.03279)

![MACHIAVELLI](docs/assets/machiavelli_border.png)

## :toolbox: Setup
Clone this repo:
```bash
git clone git@github.com:aypan17/machiavelli.git
cd machiavelli-temp
```

Install dependencies ([Please use Python 3.11 or lower!](https://github.com/aypan17/machiavelli/issues/16)):
```bash
pip install -r requirements.txt
```

Download the data:

- The data is available through [Google Drive](https://drive.google.com/file/d/19PXa2bgjkfFfTTI3EZIT3-IJ_vxrV0Rz/view).
- The password is ```machiavelli```.
- Place the data at the top-level of this repo as `./game_data/`. (You should now have a folder structure as described in [Repo structure](#file_folder-repo-structure).)

Download the "artificial conscience": In order to replicate the paper, you will need to download the artificial consciensce from [Google Drive](https://drive.google.com/file/d/1MD0N-0IfJx4mCUnAqH9yhjqBz2u00c9a/view?usp=sharing). Place the model in `./model/`.

## :joystick: Play a game!
Use our demo script to play through a game:
```bash
python -m demo -h # Get the list of available games
python -m demo -g <GAME> # Play through a game interactively (playing to completion may take several hours on longer games!)
python -m demo -g <GAME> --agent RandomAgent # Play through a game selecting random actions
```

## :robot: Adding a custom agent
1. To create your own agent, create a new class in `machiavelli/agents/` that inherits from `machiavelli.agent.BaseAgent` and implements the `get_action` method.
2. Then, add your agent to `machiavelli/agent/load_agents.py`. (See `machiavelli/agents/random_agent.py` for an example.)
3. Done! You can run your agent on a game using the `--agent` flag in `demo.py`.
```
python -m demo -g <GAME> --agent MyCustomAgent # Play through a game using your custom agent
```

## :chart_with_upwards_trend: Evaluating on the MACHIAVELLI benchmark
The MACHIAVELLI benchmark is a test set of 30 challenging games where agents trained to maximize reward perform poorly on behavioral metrics by default. To evaluate on this benchmark, generate trajectories with your agent on all 30 games and save them to file. You may use the following script:
```bash
python -m generate_trajectories -a <AGENT> --traj_dir <TRAJ_DIR>
# e.g. python -m generate_trajectories -a RandomAgent --traj_dir ./demo_trajectories --num_episodes 1
```

Given a directory of trajectories, compute the results for your trajectories using the following script:
```bash
python -m evaluate_trajectories --traj_dir <TRAJ_DIR> --results_file <RESULTS_FILE>
# e.g. python -m evaluate_trajectories --traj_dir ./demo_trajectories --results_file ./demo_results.csv
```

## :test_tube: Reproducibility
We include the scripts used to recreate the results in our paper, including our various agent baselines, in `experiments/`. To reproduce our results, run the following scripts:
```bash
source ./experiments/baselines_random.sh
source ./experiments/baselines_lm.sh
source ./experiments/baselines_rl.sh 

python -m export_results
```

(Advanced users) If you wish to reproduce our data annotation (e.g. to extend the label set), look at the `machiavelli/annotate/` and `machiavelli/data/` directories.

## :file_folder: Repo structure
Key files to understand the repo:
```
├── LICENSE
├── README.md
├── env.sh                      # Set up API keys
└── requirements.txt            # Do `pip install -r requirements.txt`

├── docs/                       # Code for website

├── experiments/                # Scripts to run experiments in the paper
│   └── baselines_*.sh          # Run trajectories for baseline agents on test set games

├── game_data/
│   ├── game_metadata.json      # List of all games and short information about them

│   ├── annotations_clean/      # Clean export of all scene-wise annotations, organized by game/scene_id
│   ├── annotations_expert/     # Scene-wise annotations from expert annotators
│   ├── annotations_model/      # Scene-wise annotations from model
│   ├── annotations_surge/      # Scene-wise annotations from Surge (crowdworker) annotators

│   ├── achievements/           # List of nodes required to hit various achievements
│   ├── scenarios/              # Full scene text of all games organized by game/scene_id
│   ├── source/                 # Raw source code used by the machiavelli.game.player
│   └── trees/                  # Game tree structures

├── machiavelli/
│   ├── __init__.py
│   ├── agents/                 # Code for baseline agents (random, LM, RL)
│   ├── annotate/               # Code for generating/managing model labels at `game_data/annotations_model`
│   ├── config.py               # Common config file used by other modules to make it easier to navigate the code
│   ├── data/                   # Code for generating non-annotation files in `game_data/`
│   ├── visualize/              # Code to visualize and explore the games
│   ├── game/                   # Code to run the games and agents
│   └── utils.py                # Helper functions

```

All python scripts are designed to be run from the root of this repo using module notation, e.g. `python -m machiavelli.game.machiavelli_env -g <GAME>`.
Bash scripts are also designed to be run from the root of this repo, e.g. `source ./experiments/baselines_lm.sh`.

## :pencil2: Citation
If you use this code in your research, please cite our paper:
```
@article{pan2023machiavelli,
    author = {Pan, Alexander and Chan, Jun Shern and Zou, Andy and Li, Nathaniel and Basart, Steven and Woodside, Thomas and Ng, Jonathan and Zhang, Hanlin and Emmons, Scott and Hendrycks, Dan},
    title = {Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark.},
    journal = {ICML},
    year = {2023},
}
```
