# How Benchmark Prediction with Fewer Data Misses the Mark

This is the code to reproduce our results in the paper. It provides all methods that we examined in the paper.

## Quick Start

```bash
pip install -r requirement.txt
```

## Example Usage

```python
python main.py --dataset_name imagenet --compress_upper_bound 50 --compress_lower_bound 50 --no-multi_process --methods ppi --num_run 1 --seed_start 0 --no-use_git
```

The supported methods are listed as follows,

- Random-Sampling: --methods random_selection
- Random-Sampling-Learn: --methods random_selection_and_learn
- Random-Search-Learn: --methods random_search_and_learn
- AIPW: --methods ppi
- PCA: --methods pca
- Lasso: --methods lasso
- Double-Optimize: --methods double_optimize
- Anchor-Points-Weighted: --methods anchor_points_weighted
- Anchor-Points-Predictor: --methods anchor_points_predictor
- PIRT: --methods pirt
- GPIRT: --methods gpirt
