# Linear Mode Connectivity on DBpedia

This document outlines the end-to-end process for preparing data, training, fine-tuning, and evaluating linear mode connectivity between models trained on the **DBpedia** dataset using **BERT**.

---

## Step 1: Download and Prepare the Dataset

```bash
bash src/dbpedia/data.sh
```

---

## Step 2: Preprocess the Dataset

Convert the raw dataset into a processed format (e.g., Parquet) suitable for training:

```bash
python src/dbpedia/preprocess.py \
    --data-dir ./data/dbpedia/ \
    --output-dir ./data/dbpedia/
```

---

## Step 3: Train the Baseline BERT Model

Train a `bert-base-uncased` model on the DBpedia dataset:

```bash
CUDA_VISIBLE_DEVICES=0 python src/dbpedia/train_model.py \
    --model-config-name bert-base-uncased \
    --train-dataset-paths data/dbpedia/train.parquet \
    --eval-dataset-paths data/dbpedia/val.parquet \
    --model-save-dir /root/weights/dbpedia/
```

---

## Step 4: Fine-tune with Reinitialized Seeds (Layer 0)

Fine-tune the pretrained model from two different initializations to enable interpolation:

```bash
CUDA_VISIBLE_DEVICES=0 python src/dbpedia/finetune_reinit.py \
    --model-path /root/weights/dbpedia/lr2e-05-epochs2-batch16/checkpoint_30115 \
    --train-dataset-paths ./data/dbpedia/train.parquet \
    --eval-dataset-paths ./data/dbpedia/val.parquet \
    --model-save-dir /root/weights/dbpedia/finetune \
    --random-seed 0
```

Repeat the above command with `--random-seed 20` to get a second model.

---

## Step 5: Naive Linear Interpolation

Interpolate between the two fine-tuned models:

```bash
CUDA_VISIBLE_DEVICES=0 python src/dbpedia/naive_interpolate.py \
    --model-a /root/weights/dbpedia/finetune/lr2e-05-epochs2-batch16-seed0/checkpoint_30115 \
    --model-b /root/weights/dbpedia/finetune/lr2e-05-epochs2-batch16-seed20/checkpoint_30115 \
    --train-dataset-paths ./data/dbpedia/val.parquet \
    --eval-dataset-paths ./data/dbpedia/test.parquet
```

---

## Step 6: Calculate Loss Barrier


```bash
python src/dbpedia/loss_barrier.py \
    --file-path results/dbpedia/[lr2e-05-epochs2-batch16-seed0+lr2e-05-epochs2-batch16-seed20].json
```

Replace the file path with the actual name of the generated interpolation result.

---



