## Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning




## Requirements

The following pakages are required to run the code:

- python==3.11.5

- pytorch==2.1.2

- transformers==4.46.1

- numpy==1.26.4

  

## Quick Start

**1. Collect the representations and compute the MI**

```bash
sh scripts/compute_mi_trajectories.sh
```

**2. Plot figures to observe the MI Peaks phenomenon**

```bash
run the plot_mi_peaks.ipynb
```



