# Belief Entrenchment

## Overview

This repository contains code for the paper on belief entrenchment.

## Installation

```bash
pip install -r requirements.txt
```

## Usage

#### Fetch data (currently forecasting data)

```bash
python -m scripts.fetch_data
```

#### Produce reasoning trajectories on a problem domain

```bash
python -m scripts.run_reasoning
```

The script will produce the files `reasoning-trajectories.json` and `bias-eval-results.json` in the run directory within the `data` folder.

Each run directory is named by `run-{RUN_ID}`. After a run is finished, you may rename the second half of the directory name to something more descriptive, and thereby change run ID. You can use the environment variable `RUN_ID=xxx` to set the run ID for the run; if a directory with the same run ID already exists, data from the previous run will be loaded and analyzed, without executing the reasoning again.

You may arbitrarily change the combination of model, problem domain, reasoning mode, and bias measurement strategy by editing the `scripts/setup.py` file. Avoid editing the main `run_reasoning.py` file except for development purposes.

#### Measure correlation between initial priors and belief updating deltas

```bash
python -m scripts.measure_correlation
```

## Codebase structure

#### `core`

Contains the core logic of the project.

- `core/algo`: Contains the debiasing algorithms (e.g. Martingale, justified flipping)
- `core/domain`: Contains the problem domains (e.g. forecasting)
- `core/policy`: Contains the policy models (e.g. API model, Local LLMs)
- `core/reasoning`: Contains the reasoning modes (e.g. Self-Debate, CoT)

Each of these components has a `schema` file that describes the components' inputs and outputs, accompanied by a number of subclass files that implement the schema for specific algorithms/domains/policies/reasoning modes.

#### `scripts`

Contains scripts for data fetching, processing, and analysis.

- `scripts/fetch_data.py`: Contains the script for fetching data from the internet.
- `scripts/measure_correlation.py`: Contains the script for measuring the correlation between initial priors and belief updating deltas.
- `scripts/run_reasoning.py`: Contains the script for producing reasoning trajectories on a problem domain.

In addition, we have

- `scripts/setup.py`: Contains the parameters that other scripts depend on.

#### `utils`

Contains utility functions for the project.

- `utils/io_utils.py`: Contains the utility functions for input/output operations, including the handling of JSON formatting.
- `utils/debate_processing_utils.py`: Contains the utility functions for processing the debate data.
- `utils/forecasting_processing_utils.py`: Contains the utility functions for processing the forecasting data.
- `utils/path_utils.py`: Expands the PATH variable to include all levels of the project directory.