# How Sampling Impacts the Robustness of Stochastic Neural Networks

This repository is the official implementation of the paper "How Sampling Impacts the Robustness of Stochastic Neural Networks". 

## Requirements

To install requirements:

```setup
pip install -r requirements.txt
```
Please set the path to the repository in the config.py file accordingly to your setting.
## Training

To train the model(s) in the paper, run for example this command for training SIN with noise 0.1 :

```train
python 1_train_models.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1
```
All trained models on FashionMNIST are saved under models/{dataset}.

## Evaluation

To create adversarial examples with FGM on the logits for different amounts of samples, run for example:

```eval
python 2_create_adversarials.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='strong' --attack_samples=1
python 2_create_adversarials.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='strong' --attack_samples=5
python 2_create_adversarials.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='strong' --attack_samples=10
python 2_create_adversarials.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='strong' --attack_samples=100
```
The accuracies for different perturbation strengths eps are saved under ../results in a file called "r_value_adv_1000_FashionMNIST_SNN_strong_1_2.5_{eps}_0_100_0.1_11999".

To evaluate the angle for the different attack strength, run
```eval
python 4_run_linear_conditions.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='FGM' --attack_samples=1 --n_samples=100
python 4_run_linear_conditions.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='FGM' --attack_samples=5 --n_samples=100
python 4_run_linear_conditions.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='FGM' --attack_samples=10 --n_samples=100
python 4_run_linear_conditions.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 --attack='FGM' --attack_samples=100 --n_samples=100
```
the respective results are saved in a pandas dataframe results/r_value_adv_1000_full_FashionMNIST_SNN_strong_100_2.5_{eps}_0_100_1_17999'.
The results reported in the table in the main paper can be generated by e.g.
```eval
python 5_eval_accuracy_wrt_inference_samples.py --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 
python 6_eval_gradient_variances --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1 
python 7_eval_prediction_variance --net='SNN' --dataset='FashionMNIST' --stoch_varianz=0.1
```
the results are saved under ../results/CLT_Var_FashionMNIST_SNN_0_0_0.1_10999 for example.

## Results
Since our work is about stochastic neural networks we fixed the random seed where appropriate to be able to generate the same results.
However, the general observed tendencies: more attack samples lead to an accuracy decrease, more inference samples do not affect the accuracy, the angle is decreased with more samples and so is the gradient norm, are independent of the exact numerical values.
