# CLIP Adversarial Attack Results Analysis Tools

This directory contains tools for gathering, analyzing, and visualizing results from CLIP model adversarial attack experiments.

## Tools Overview

1. `gather_results_csv.py` - Python script to extract data from JSON result files and compile into CSV format
2. `results_analysis.ipynb` - Jupyter notebook for visualizing and analyzing the results

## Usage Instructions

### 1. Gathering Results

The `gather_results_csv.py` script processes JSON files in a directory and extracts key information, focusing on:

- Algorithm type (FARE or LORE)
- Training epsilon values
- Dataset information
- Attack parameters
- Accuracy metrics (primarily acc1)

To run the script:

```bash
python gather_results_csv.py --input_dir /path/to/results/directory --output_csv output_results.csv
```

Default values:
- `input_dir`: `/YOUR_ROOT_PATH/CLIP_benchmark/results/ViT-B-32/final`
- `output_csv`: `/YOUR_ROOT_PATH/CLIP_benchmark/results_analysis.csv`

The script generates:
1. A detailed CSV with all extracted data
2. Summary CSVs for clean evaluations
3. Summary CSVs for adversarial evaluations, grouped by attack parameters

### 2. Analyzing Results

After running the gathering script, you can use the Jupyter notebook for visualization:

```bash
jupyter notebook results_analysis.ipynb
```

The notebook includes:
- Data loading and overview
- Algorithm comparison visualizations
- Training epsilon analysis
- Attack epsilon analysis
- Dataset-specific performance analysis
- Advanced visualizations comparing robustness across configurations

## File Naming Convention

The analysis relies on extracting information from standardized filenames:

- Files starting with `clean_` are clean evaluations
- Files starting with `adv2_` are adversarial evaluations
- Algorithm type (FARE or LORE) is extracted from the filename
- Training epsilon is extracted from patterns like `eps1`, `eps2`, etc.
- Attack parameters are extracted from the JSON content and filename

## Example

For a file named `adv2_ViT-B-32_ViT-B-32_openai_imagenet_l2_pgd_FARE_base_eps1_vnbyu_ViT-B-32_openai_imagenet_l2_pgd_FARE_base_eps1_vnbyu.pt_caltech101_1000_bs50_aa_4_100.json`:

- Algorithm: FARE
- Training epsilon: 1
- Dataset: caltech101
- Attack: aa (AutoAttack)
- Attack epsilon: 4.0 (from JSON content)
- Accuracy (acc1): extracted from the metrics field in JSON

## Logging

The script logs its progress and any errors to:
1. Console output
2. A log file named `gather_results.log`

## Requirements

- Python 3.6+
- pandas
- matplotlib
- seaborn
- numpy
- Jupyter (for running the notebook)

## Notes

- If certain fields are missing in the JSON files, the script will handle them gracefully
- For the best visualization experience, ensure you have a recent version of matplotlib and seaborn installed 