# Requirements

```
accelerate==0.3.0
python=3.7.10
pytorch=1.9.0
torchaudio=0.9.0
torchvision=0.10.0
deepspeed==0.5.0
scikit-learn==0.24.2
scipy==1.7.0
tokenizers==0.10.3
tqdm==4.61.2
huggingface-hub==0.0.13
datasets==1.9.0
pandas==1.3.0
```

# Usage

First, you should `cd scripts` .

After that, to **reproduce** our all results, just use:

```
bash roberta_base_all.sh
bash roberta_large_all.sh
```

Or you can reproduce the results of a particular dataset individually.

For example, `RoBERTa_Base` + `CoLA` :

```
export num_gpus=2
export CUBLAS_WORKSPACE_CONFIG=":16:8" # https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility
export PYTHONHASHSEED=0
export output_dir="./cola"
export CUDA_VISIBLE_DEVICES=0,1
python -m torch.distributed.launch --nproc_per_node=$num_gpus \
examples/text-classification/run_glue.py \
--model_name_or_path roberta-base \
--task_name cola \
--do_train \
--do_eval \
--max_seq_length 512 \
--per_device_train_batch_size 32 \
--learning_rate 5e-3 \
--num_train_epochs 100 \
--output_dir $output_dir/model \
--overwrite_output_dir \
--logging_steps 10 \
--logging_dir $output_dir/log \
--evaluation_strategy epoch \
--save_strategy epoch \
--warmup_ratio 0.06 \
--apply_lora \
--lora_r 2 \
--lora_alpha 4 \
--seed 0 \
--weight_decay 0.1
```

# Acknowledge

This code is based on [LoRA](https://github.com/microsoft/LoRA), we thank for their efforts.