# Skill-driven Neurosymbolic State Abstractions


## Installation
Create a new virtual environment using conda:
```
conda create -n ns-mdp python=3.9.20 setuptools=65.5.0 wheel=0.38.4 pip=24.0
conda activate ns-mdp
pip install -r requirements.txt
```

## Experiments

Run the MNIST ChainWalk experiment for a chain of length 6
```bash
python exp_chain.py \
    --grid-size 6 \
    --eps 0.05 \
    --min-t 1.5 \
    --t-delta 0 \
    --n-cluster 5 \
    --min-samples 10 \
    --n-sample 1000 \
    --seed 0 \
    --save-folder mnist_chain__size_6__nsample_1000__seed_0 \
    --name mnist_chain
```
or length 10
```bash
python exp_chain.py \
    --grid-size 10 \
    --eps 0.05 \
    --min-t 1.5 \
    --t-delta 0 \
    --n-cluster 5 \
    --min-samples 10 \
    --n-sample 10000 \
    --seed 0 \
    --save-folder mnist_chain__size_10__nsample_10000__seed_0 \
    --name mnist_chain 
```
which will output a visualisation of the abstract states in `chain.pdf`.

To run the MNIST GridWalk experiment for a grid size of (6, 6), first train MSA embeddings
```bash
python train_msa.py \
    --env mnist \
    --msa-config mnist.yaml \
    --grid-size 6 6 \
    --eps 0.05 \
    --val-split 0.1 \
    --n-sample 10000 \
    --seed 0 \
    --save-folder expname__nsample_10000__seed_0 \
    --name mnist_msa
```
then build the abstract MDP using those
```bash
python exp_mnist.py \
    --grid-size 6 6 \
    --eps 0.05 \
    --min-t 0.1 \
    --t-delta 0.5 \
    --n-cluster 5 \
    --min-samples 10 \
    --msa-encoding \
    --use-msa-ind \
    --n-sample 10000 \
    --seed 0 \
    --save-folder expname__nsample_10000__seed_0 \
    --name mnist_msa 
```
and test
```bash
python test_mnist.py mnist_msa__nsample_10000__seed_0 mnist_msa__nsample_10000__seed_0
```

Similarly, to run the VisualMaze experiment at 80 x 60 resolution, first train MSA embeddings
```bash
python train_msa.py \
    --env maze \
    --msa-config maze.yaml \
    --res 80 60 \
    --val-split 0.1 \
    --n-sample 5000 \
    --seed 0 \
    --save-folder expname__nsample_5000__seed_0 \
    --name maze_msa
```
build the abstract MDP
```bash
python exp_maze.py \
    --res 80 60 \
    --n-cluster 5 \
    --min-sample 10 \
    --min-t 0.01 \
    --use-msa-ind \
    --n-sample 5000 \
    --seed 0 \
    --save-folder expname__nsample_5000__seed_0 \
    --name maze_msa
```
and test
```bash
python test_maze.py maze_msa__nsample_5000__seed_0 maze_msa__nsample_5000__seed_0
```

Run the Montezuma's Revenge experiment
```bash
python exp_monty.py \
    --clip-eps 0.9 \
    --t-delta 0.1 \
    --min-t 0.1 \
    --n-cluster 5 \
    --min-samples 10 \
    --n-sample 10000 \
    --seed 0 \
    --save-folder monty__nsample_10000__seed_0 \
    --name monty
```

For comparison, train a goal-conditioned DQN with hindsight experience replay on MNIST grid
```bash
python gcdqn_mnist.py \
    --grid-size 6 6 \
    --eps 0.05 \
    --n-sample 10000 \
    --seed 0
```
or on Visual Maze
```bash
python gcdqn_maze.py \
    --res 80 60 \
    --n-sample 10000 \
    --seed 0
```

Save folder naming is as follows:
```bash
# The save file will be under <myname>__<arg1>_<value1>__<arg2>_<value2> in this setting
--save-folder <experiment_name>__<arg1>_<value1>__<arg2>_<value2> --name <myname>
```
In other words, the first part in the `--save-folder` is a temporary experiment name while `--name` is used in concatenation with the rest of the arguments.